An AI can apply your written rule to every answer, but only after you check it agrees with you on examples it has never seen. Then you can fix the product and measure it.
Brief an AI reviewer like a new team member
Labelling 116 gaps by hand took real time. Doing it after every product change is not realistic. So the next step is to write the rule as instructions for an AI model and let it label the gaps.
The first version of the AI reviewer caught 22 of the 50 assumed gaps it was tested on. Reading the ones it missed showed the rule was fuzzy in one place: it did not say clearly what counted as "seen". The expert's own labelled examples filled that in, and the next version caught 34 of 50. A third version closed a loophole, where a guess was written up as a critique of something visible, and caught 45 of 50.
When an AI reviewer gets it wrong, check your rule before you blame the reviewer. A vague brief produces vague work from any new team member.
An AI model that scores outputs against your rubric is an LLM-as-judge, also called a judge, grader or evaluator. Rewriting its instructions is prompt iteration. Checking its verdicts against yours is judge validation or alignment.
Keep some examples back
Those three scores all came from the same half of the gaps. Before the first version ran, the 116 labelled gaps were split by screenshot into a tuning half and a held-back half. Nobody looked at the held-back half while rewriting the reviewer. When the third version was finished, it was frozen and run once on the held-back half.
- Version 1, tuning half
- Caught 22 of 50 assumed gaps. Raised 2 false alarms in 11 fine gaps.
- Version 2, tuning half
- Caught 34 of 50. Raised 1 false alarm in 11.
- Version 3, tuning half
- Caught 45 of 50. Raised 2 false alarms in 11.
- Version 3, held-back half
- Caught 41 of 44. Raised 5 false alarms in 11.
On new examples the reviewer still caught almost every guess. It also raised far more false alarms: it called 5 of the 11 fine gaps assumed. The tuning half had hidden that. Without the held-back half, the study would have trusted a reviewer that looked close to perfect.
Think of the tuning score as a prototype demo on the happy path you rehearsed. The score that counts comes from examples the reviewer has never seen, measured once.
The half you tune on is the dev set (sometimes train/dev). The held-back half is the test set or holdout, and the practice is a train/test split. A reviewer that only works on the examples you tuned it with is overfitting.
Report counts instead of percentages
"Caught 41 of 44" tells the reader how few examples there were. "93%" sounds more certain than 44 examples can support. With only 11 fine gaps in the held-back half, one extra false alarm moves the rate by nine points. The FAQ suggests 100 to 200 labelled examples per failure type before trusting an AI reviewer on its own. The study had 116 in total, which was enough to learn the method. It was not enough to stop checking the reviewer's work.
"Caught 41 of 44 real problems" is the true positive rate, also called recall. "Raised 5 false alarms in 11" is the false positive rate. Its flip side, correctly passing the good ones, is the true negative rate. How many examples you have is the sample size.
Fix the product, then run the same check again
This is where the eval pays off. Change the product, run the same screenshots through it, let the frozen AI reviewer count the problem, and see whether the number moved.
Fix 1 told the audit to turn guesses into "Worth checking" questions instead of gaps. The AI reviewer then flagged 86 of 115 gaps as assumed, against 93 of 116 before. The number had hardly moved. The fix also broke 2 of the 41 audits, which came back as unreadable data that a real user would have seen as an error.
Reading the misses explained the weak result. The audit's instructions still contained examples shaped like the failure, such as "has no thumbs-up/down". The rule said one thing and the examples showed another, and the examples won. When you brief an AI, show it what good looks like.
Fix 2 replaced those examples. Assumed gaps fell to 73 of 107, honest gaps rose from 23 to 34, and all 41 audits ran cleanly.
Test the fix on fresh examples
Those 41 screenshots were the ones everyone had been staring at while writing the fix. So the study ran the original and the fixed audit on 19 fresh screenshots nobody had looked at. On the original audit, 60 of 65 gaps were guesses. On the fixed audit, 50 of 57 were. That change is about the size of normal run-to-run variation.
Most of fix 2's improvement on the 41 came from the wording fitting those 41. The same lesson as the held-back half applies to product fixes: check the change on examples you did not use to make it. When rewording stops moving the number on fresh examples, the next step is usually a structural change, such as an automatic second check on every gap, and that costs time and money of its own.
Check what the fix might have broken
The AI reviewer counts one failure type. A fix can break something else without that count noticing. So after fix 2, the expert reviewed all 41 audits again with the original question.
The result was 33 pass and 8 fail, against 29 and 12 on the first review. The expert also judged the new "Worth checking" questions, 148 of them, to be helpful to users. That decision had a design consequence: the fix could not ship until the results screen had a place to show those questions.
The same loop measured two more fixes:
- Confidence on ordinary chat. The expert first marked 20 screens yes or no for whether knowing the AI's certainty would change what a user does. After a fix, the audit asked for Confidence Visualization where it was not needed on 0 screens, down from 8.
- Marketing images. A fix to recognise marketing pictures still gave AI gaps to 6 of 12 marketing images, and it wrongly called 4 of 28 real product screens marketing. It hurt real screens, so it was reverted.
Re-running the same examples after every change, to catch anything that got worse, is a regression eval or regression test. Fresh examples that played no part in the fix are an out-of-sample check. A person skimming the outputs before release is a human spot-check.
Tools you can use
You can run this whole method with tools a designer already knows. These were the options at the time of writing, October 2026:
- A spreadsheet with an AI column
- One row per answer, a pass or fail column for you, an AI column for the reviewer, and a pivot table to compare the two. Reading screenshots needs an image-capable add-on or a paid Airtable plan, so check that before you start.
- A review page an AI builds for you
- What the study used. A single HTML page built with Claude Code or a similar tool, with images kept on your own machine. The AI reviewer is a short script, so you need an AI coding tool to write and run it.
- Braintrust
- The one vendor whose documentation covers uploading existing outputs, reviewing them, and passing screenshots to an AI judge. Comparing the judge with your labels means exporting both and counting yourself.
- Langfuse
- A keyboard-driven review queue and a built-in view of how often the judge agrees with you. Its no-code path does not pass images to the judge, so it suits text-only features.
Most eval tools assume an engineer has already connected the live AI feature. If your answers already exist as files, check that the tool can import them, and that it can show the AI reviewer any images, before you commit. OpenAI's hosted Evals dashboard is scheduled to shut down on 30 November 2026, so avoid starting new work there.
Glossary: plain words and engineering terms
- Checking AI output against a standard
- Evaluation, eval, evals
- The saved record of one AI answer
- Trace, log
- The examples you test on
- Dataset, eval set
- Messy and irrelevant examples
- Edge cases, negative cases
- Testing the whole journey
- End-to-end evaluation
- The review screen
- Annotation tool, labelling interface
- Marking pass or fail with a note
- Annotating, labelling
- Sticky notes on what is wrong
- Open coding
- Affinity-mapping the notes
- Axial coding
- The named groups of problems
- Failure modes, failure taxonomy
- The whole notes-then-groups process
- Error analysis
- How often two reviewers agree
- Inter-rater agreement, Cohen's kappa
- The expert who makes the final call
- Domain expert, SME, benevolent dictator
- The written standard
- Rubric, criteria
- Your standard sharpening as you go
- Criteria drift
- The settled verdicts
- Ground truth, gold labels
- Judging each piece separately
- Finer granularity, per-item labelling
- Yes or no instead of 1 to 5
- Binary evaluation instead of a Likert scale
- An AI that applies your standard
- LLM-as-judge, grader, evaluator
- Improving its instructions
- Prompt iteration
- Checking it agrees with you
- Judge validation, alignment
- The half you tune on and the half you keep back
- Dev set and test set, holdout
- Only working on the examples you looked at
- Overfitting
- "Caught 41 of 44 real problems"
- True positive rate, recall
- "Raised 5 false alarms in 11"
- False positive rate
- Running the same check after every change
- Regression eval
- AI Evals: Everything You Need to Know· Hamel Husain and Shreya Shankar
The evals FAQ this course draws on: 60 questions covering error analysis, judges, sample sizes and tooling.
- Should I build a custom annotation tool or use something off the shelf?· Evals FAQ
The case for designing your own review screen, and when a ready-made tool fits better.
- Why do you recommend binary pass/fail evaluations instead of 1-5 ratings?· Evals FAQ
Why yes or no questions produce more consistent verdicts than rating scales.
- How many examples do I need for an eval?· Evals FAQ
Sample sizes for finding problems, for checking an AI judge, and for a repeatable test set.
- How do I know if I can trust my automated eval?· Evals FAQ
Checking a judge against human labels, and spotting when it has overfit to the examples you tuned it on.
You can run the whole loop
- Brief an AI reviewer from your rule and check it on held-back examples
- Report results as counts
- Measure each fix on the same examples, then on fresh ones
- Re-review by hand for anything the count cannot see
Course complete. Pick one AI feature you work on, save 20 to 50 real answers, and review them with one yes or no question.