Notes, Clusters and a Second Opinion

Write plain notes first, cluster them the way you would affinity-map research findings, then let a second reviewer show you what you missed.

Sticky notes first, categories later

When you review an answer, write down the first thing that is wrong in plain words. Do not sort the notes into categories yet. Categories chosen in advance make you look for the problems you already expected and skim past the rest.

The FAQ recommends reviewing at least 30 examples yourself before you let an AI suggest problems, because early suggestions can steer your judgement. It also asks you to note problems that are not the model's fault, such as a missing feature or a confusing layout. Some of those turn out to be design fixes that need no eval at all.

Then cluster them

Once the notes are in, group them the way you would group findings from interviews. Name each group, then count how many failed audits fall into it. Counting matters, because it tells you which problem to work on first.

In the study, the notes fell into six groups. One audit can sit in more than one group, so the counts add up to more than the number of failed audits.

States guesses as findings (7)

The audit reports a gap it cannot see in one screenshot, such as a missing hover state, what is below the scroll, or whether undo exists, and presents it as fact.

Confidence on ordinary chat (7)

It asks for confidence scores and explanations on normal chat replies, where nothing is being researched and certainty does not change what the user does next.

Wrong call on whether AI is there (5)

It misses real AI, scoring Claude Code and an AI email summary at zero, or invents AI on a government search page and a marketing hero.

Audits pictures of a product (4)

It treats marketing images, a documentation illustration and a near-empty welcome pop-up as a working interface.

Misreads the screen (3)

It quotes text too small to read and calls a parked car "moving" on a car dashboard.

Fixes without context (1)

It recommends adding things that may already exist elsewhere in the product, or that would add clutter.

The biggest group, "states guesses as findings", was not on anyone's list before the review started. That is the reason to write notes before choosing categories. The FAQ calls this activity "the most important activity in evals".

Get a second reviewer

One reviewer is a sample of one. The study added a second reviewer: a separate AI model that went through the same 41 audits with the same question and never saw the first reviewer's verdicts. It passed 24 and failed 17.

The two reviewers agreed on 28 of 41 audits. The interesting part was what each one caught:

• The domain expert caught judgement problems. Gaps that were guessed instead of seen, and screenshots that were not a working product at all.
• The second reviewer caught accuracy problems. A car dashboard audit that called a parked car "moving", two real AI features scored as having no AI, and a government homepage treated as an AI product.

Both kinds were real failures. A second reviewer is useful mainly because it reads differently from you. A colleague, a researcher or an AI model can play the part, as long as they review without seeing your verdicts first.

The expert makes the final call

A second reviewer does not get an equal vote. The domain expert ruled on each of the 13 disagreements, the way a design lead closes out a critique. Sometimes the expert sided with the second reviewer, and sometimes with their own first verdict. The final result was 23 pass and 18 fail. Those rulings became the answer key that everything later in the study was measured against.

The FAQ recommends the same setup for most small and medium teams: one trusted expert who makes the quality call. If you feel you need five experts to judge a single answer, the FAQ suggests the product scope may be too broad.