Write the Standard Down

Your standard will sharpen as you review. Write it down as a yes or no rule, and judge the smallest piece of an answer that can be wrong.

Pick one failure to work on

Six failure types is too many to fix at once. If you change several things together, you cannot tell which change helped. The study picked one: "states guesses as findings". It was tied for the most common, it came from the expert's own notes, and it was the hardest of the six to check.

Turn it into a rule

A failure type is a label. A rule tells someone else how to apply it. The study settled on this one:

Notice that the rule is a yes or no question. Each gap is either a critique of something visible, or a guess about something that is not. There is no "somewhat assumed".

Ask yes or no questions instead of 1-to-5 ratings

The old audit check scored each answer 1 to 5 on five qualities. One of its three examples was a plain chat screen whose own written notes said Confidence Visualization and Explainable AI did not apply there. The automated checks could not fail on that, and the 1-to-5 scorer never flagged it. A rating gives you a number like 3.6 that is hard to act on. A yes or no question forces a decision and tells you exactly which answers to look at.

The FAQ makes the same case: "Binary evaluations force clearer thinking and more consistent labeling." If you want to track partial progress, split the quality into several yes or no checks and count how many pass.

Judge the smallest piece that can be wrong

Judging a whole answer at once hides problems. You note the first thing wrong and skim the rest. So the study switched from judging each audit to judging each gap inside it: 116 gaps across the 41 audits, each marked "seen" or "assumed" by the expert before any AI reviewer was involved.

The difference is the one between "this screen feels off" and a line-by-line design review. A whole-answer verdict tells you something is wrong. A per-piece verdict tells you how much is wrong and where.

Expect your standard to move

Here is what happened when each gap was checked against the written rule. 94 of the 116 gaps were assumed. Yet in the first whole-audit review, the same expert had passed 29 of 41 audits, and many of those audits were full of exactly these gaps.

The audits had not changed. The standard had become clearer. Reviewing the first batch is what taught the expert what the standard should be, and writing it down made it possible to apply it evenly.

Shreya Shankar and her co-authors describe this effect in their research: "users need criteria to grade outputs, but grading outputs helps users define criteria." It is normal, and it is the reason the FAQ advises against writing a rubric before you have looked at real answers.