How to test an AI feature before you trust it

An AI feature is not tested by using it a few times and liking the answers. Liking cannot be repeated, handed to a colleague, or compared with next month's version, and it says nothing about the questions you did not think to try. What replaces it is a written question set, an answer key two people agree on, and a decision about where a wrong answer lands.

Why write the questions before the build?

Because the set is the specification, and the only artefact that survives a change of model or supplier.

Written early, it decides scope: many questions need a record the feature will not have, and finding that in a document costs an afternoon instead of a launch. Written late, it measures what was built rather than what was needed — whoever writes it has seen the answers, so the questions drift towards the ones that work.

What are the twenty questions?

Twenty is a starting number, not a target: enough to cover the real shapes, few enough for one person to mark by hand. Take them from where people already ask: the support inbox, the salesperson's phone. Most will be dull, the same few questions reworded, and that majority pays for the feature. Then add four kinds nobody volunteers:

• the question the data cannot answer, where the only correct reply is that it is not there;
• the question about a person rather than a record, which the feature should decline;
• the question typed the way people type: dialect, abbreviation, two questions in one message;
• the question that tries to reach another customer's record.

How do you write a correct answer when there is no single right one?

You do not write the answer. You write what a good answer must contain, what it must not contain, and what it must do when it cannot answer, then two people mark the same set and compare.

For "why is my order late", a good answer names the current status and the next step. It gives no delivery date, blames no courier, and apologises for nobody. That is not a sentence to match; it is a rubric, and the rubric is what makes marking repeatable.

What you score	A pass	A fail
The required fact	The status from the record, sourced	A plausible figure, no source
The forbidden claim	No date, no price, no promise	"It should arrive soon"
The refusal	Names the gap, offers the next step	Answers a neighbouring question
The route	Reaches whoever can answer	Stops at "contact us"

Where the two markers disagree, the criterion is not yet written. That disagreement is the most useful output, and averaging it away is the mistake: a boundary nobody can state is a boundary nobody can hold.

Why measure the failure that costs money, not the average?

Because an average hides the only failures that matter. Split the set by consequence before counting: questions where a wrong answer costs a repeated question, and questions where it costs money, a promise, or a customer's data. The first can be wrong occasionally and still be worth having. The second cannot be wrong at all, and no overall figure makes it acceptable. A feature that is fine on the easy majority and wrong on the refund question is not mostly good; it is unfit for the question it will be blamed for.

Who marks the answers, and what does the marking produce?

Someone who does the job, marking blind, with the builder's expectation out of view. The builder's own pass catches obvious breakage; it is not evidence.

The output is a labelled set — the question, what a good answer must contain, the answer as it came, a verdict, and one line of why. NIST's AI Risk Management Framework treats measurement as part of the system rather than a phase after it, and this file outlives every model decision you make.

What if it is right most of the time?

Then the question stops being whether it works and becomes where the wrong ones land. Three devices, decided per question type rather than per feature:

The threshold. Some questions may be answered automatically, some only drafted for a person to send, some must go straight to a person — written as a list a developer can implement.

The fallback. What the customer receives instead, written by a person and reviewed like any customer message. A spinner and an apology is an outage, not a fallback.

The sample. Twenty logged exchanges read weekly, drawn at random rather than from the ones that looked strange. The failure that reaches a customer is usually the one nobody selected.

What actually stops a confident miss from reaching a customer?

Something that does not depend on the model's judgement. A lookup for the facts that must be exact, so the figure comes from the record. A check that the page an answer cites still says that. And a hard rule that anything below the bar becomes a handover rather than a hedged reply, because hedging survives review: "approximately" reads as careful and passes a reviewer who is skimming.

The OWASP Top Ten keeps broken access control as its own category, which is why the fourth kind of question above exists: an assistant with a retrieval layer can be asked for a record it should never return.

How do you run the test in order?

1. Collect twenty real questions from the log and the phone, in the asker's words.
2. Write what a good answer must contain, must not contain, and must do when it cannot answer.
3. Split by cost of being wrong: which questions are answered, drafted, or handed over.
4. Mark the current human process on the same set; keep it as the baseline.
5. Build, then mark blind with someone who does the job.
6. Rewrite the criteria wherever the markers disagreed, and mark again.
7. Write the threshold and the fallback as text a developer implements.
8. Re-run the same set after every change, and add the questions users asked this month.

Where this goes wrong

The set written by the builder. Every question is one the feature already answers, the marks look good, and production's first week supplies the missing ones.

The threshold set once. A column is renamed upstream, the answers drift, and the review scheduled for the next code change never happens.

The set that never grows. Real users ask new things every month, and a set written once measures the month it was written.

What this does not cover

Nothing here is about choosing a model, writing instructions or estimating cost per query; those decide how good the feature could be, and this decides whether you can tell. It is not a security review or a legal sign-off, and it assumes someone can say what a correct answer looks like — when nobody can, the evaluation proves the specification was missing.

We write the question set and the answer key before the build, and keep them beside the code so every later change is measured against the same twenty questions.