Pick your test examples the way you would recruit research participants, then design a review screen that makes judging each answer fast and fair.
Recruit examples like participants
You would not run a usability study with only your happiest customers. A test set deserves the same care. If every example is one the feature handles well, the check can only tell you it works.
The old audit check had three hand-picked chat screens. Real users mostly upload dashboards and chat threads, according to the site's own admin page. So the study built a set of 41 screenshots from four sources:
- 3 screenshots from the old check, kept so the two could be compared
- 13 stills from the pattern library
- 14 frames taken from screen recordings of real AI products
- 11 live pages captured fresh, including pricing, login and legal pages, a government homepage, dashboards and a mobile AI chat
Two kinds of example matter more than they look. The first is pages that should get no AI advice at all, such as a pricing page or a login form. If the tool invents AI problems there, you want to know. The second is messy captures. Some live pages came back with a cookie banner covering the screen, a welcome pop-up, or a "Dashboard not found" message. The study kept them, because real uploads are messy too.
One trap showed up early. The pattern-library images were chosen to show patterns done well, so a set made only from them leans positive. Mixing sources keeps the set closer to what users actually bring.
The fixed collection of examples you test against is a dataset or eval set. Messy and awkward examples are edge cases. Examples that should produce nothing, like the pricing page, are negative cases.
Hamel Husain and Shreya Shankar's FAQ suggests a minimum setup of 30 minutes reviewing 20 to 50 outputs whenever you make a significant change. That is a good first target. The study used 41.
Design the review screen like any other tool
Reviewing 41 audits by scrolling through raw data is slow and tiring, and tired reviewers make inconsistent calls. So the study built a small review page, and it was designed with the same care as any internal tool:
- The input and the output side by side. The screenshot on the left, the audit on the right, laid out the way a user sees it.
- One question. "Would you be happy if a real user got this audit?"
- A yes or no answer. Pass or fail, plus a short note on the first thing that is wrong.
- One keystroke per action. P for pass, F for fail, N to write a note, arrow keys to move on.
- A progress bar and autosave, so a review can be paused and picked up later without losing anything.
The FAQ is direct about this part: "Build a custom annotation tool. This is the single most impactful investment you can make for your AI evaluation workflow." Of the whole process, it is the step that looks most like ordinary design work. Show the output in the form the user sees it, keep all the context on one screen, ask one clear question, and make the next item one keystroke away.
You do not need an engineer for this. The study's page is a single HTML file built with an AI coding tool, and the FAQ notes that tools like these can build a tailored review interface in hours. Lesson 5 covers ready-made tools if you would rather not build one.
A review screen is an annotation tool or labelling interface. Marking each answer pass or fail is annotating or labelling, and the verdicts you produce are labels.
What the first review found
The domain expert reviewed all 41 audits with the page: 29 passed and 12 failed. Every failure came with a note in plain words, such as "all of the gaps here are assumed". Those notes are the raw material for the next lesson.
You can set up a review
- Build a test set that mirrors real use, including messy and irrelevant examples
- Design a review screen around one yes or no question
- Write a short note on the first thing wrong with each failure
Next: turning those notes into named problems, and getting a second opinion.