Here is the problem that catches every team shipping their first AI feature. You launch, adoption looks healthy, engagement is fine, and you have no idea whether the thing is producing good output or confident nonsense. Your usual instrumentation cannot see quality, only usage.
Worse, you cannot safely change anything. Swap the model, edit the prompt, adjust the retrieval, and you might have improved it. Or silently broken a case that worked last week. Without evaluation you are shipping blind, and the blindness compounds.
What an eval actually is
An eval is a test suite for probabilistic output. A set of inputs, a definition of what good looks like, and a way of scoring how close you got. Conceptually it is unit testing; the difference is that the correct answer is a range rather than a value.
The three layers, cheapest first:
| Layer | What it checks | Cost | Catches |
|---|---|---|---|
| Deterministic checks | Format, length, required fields, forbidden content, valid JSON | Near zero | Structural breakage, obvious safety failures |
| Model-graded | Another model scores the output against criteria | Low | Relevance, tone, whether the question was answered |
| Human review | A person judges a sample against a rubric | High | Everything else — and it calibrates the other two |
Run all three. Deterministic checks on every request in production. Model-graded evals on every change before it ships. Human review on a sample, weekly, forever — because model-graded evals drift and need recalibrating against human judgement.
The golden set
The foundation is a fixed collection of representative inputs with known-good outputs. Fifty is enough to start; a few hundred is comfortable. Building it is unglamorous and it is the highest-value week of work on any AI feature.
What goes in it:
- The common cases — whatever 80% of real traffic looks like.
- The edge cases you know about — empty input, enormous input, wrong language, adversarial phrasing.
- Every bug you have ever fixed. This is the part teams skip and regret. A fixed case that is not in the golden set will break again.
- Cases where the right answer is "I don't know." A model that never declines is a model that will confidently invent.
Deciding what good output looks like is a product decision — it encodes tone, risk appetite, and what the user actually needed. Engineers can build the harness and run it. They should not be the ones deciding whether a hedged answer is better than a wrong one. If you delegate the rubric, you have delegated the product.
Writing the rubric
"Is the output good?" is unanswerable. Break it into dimensions that can be scored independently, because a change that improves one usually costs another.
- Correctness — is it factually right, given the source material?
- Groundedness — is every claim supported by the retrieved context, or did it invent something plausible?
- Relevance — did it answer the question actually asked?
- Completeness — did it miss something important?
- Tone and format — does it match the product's voice and structure?
- Safety — refusals where refusals are appropriate.
Score each on a small scale — 1 to 3, or pass/fail — and track them separately. An aggregate "quality score" hides exactly the trade-off you need to see: the prompt change that made answers more complete and less grounded is a change you probably do not want.
The regression problem
Traditional software mostly changes when you change it. AI features change underneath you. A provider updates a model, retrieval content shifts, user behaviour moves — and yesterday's working feature quietly degrades with no deploy and no error.
Three habits that contain this:
- Pin model versions where the provider allows it, and treat an upgrade as a change requiring a full eval run.
- Run the golden set on a schedule, not only on deploy. Weekly is usually enough to catch drift before users do.
- Alert on eval score, the same way you alert on error rate. A five-point drop should page somebody.
Someone changes the prompt, tries three examples by hand, decides it looks better, and ships. Two weeks later a customer reports a case that used to work. Nobody can say which change broke it, because there is no record of what worked before. Every AI team arrives here; the ones that build a golden set early spend a week, and the ones that do not spend a quarter.
Production signals worth watching
Evals tell you about quality offline. These tell you about reality:
- Edit rate. What share of generated output does the user change before using it? The single best quality proxy in most products.
- Regeneration rate. Asking again means the first answer failed.
- Abandonment. Generated, then nothing. Usually worse than an explicit thumbs-down.
- Fallback usage. How often people take the manual escape hatch instead.
- Explicit feedback. Useful, but the volume is tiny and skewed. Treat it as a source of examples for the golden set, not as a metric.
Note that thumbs-up rate is the metric teams instinctively reach for and the weakest on this list. A fraction of a percent of users click it, and they are not a random fraction.
A workable starting setup
- Fifty golden cases, written by you, covering common and edge cases.
- A rubric with four or five independently-scored dimensions.
- Deterministic checks running on every production request.
- A model-graded eval run on every prompt or model change, blocking release on regression.
- Twenty human-reviewed samples a week, feeding new cases back into the golden set.
- Edit rate and regeneration rate on the dashboard next to adoption.
That is perhaps a week of setup and an hour a week to maintain. It is the difference between being able to improve a feature and only being able to change it.
For the prior question of whether the feature should exist, see when AI is the wrong answer. For what it costs to run, see AI feature unit economics.