The Study Behind This Course: Running an Eval on an AI Audit Tool

This course was written from one real eval, run on the AI audit tool on this site. The tool looks at a screenshot of an AI product and lists the design patterns it is missing. This page is the study itself: what was tested, what broke, what the fixes changed, and what is still wrong.

Why run it

I wanted to see what I could learn about evals as a designer, and I wanted to test the tool extensively rather than trust a handful of good-looking audits.

The method follows Hamel Husain and Shreya Shankar's evals FAQ, done by hand where possible, with engineering help only for saving the data, running the tool at scale and wiring up the AI reviewer. The whole study cost about $30 in model calls.

Step one: read every answer yourself

The tool was run on 41 real screenshots, deliberately including messy ones: pricing pages, login screens, legal text, dashboards with no AI in them at all. I gave each audit one verdict, pass or fail, against a single question: would you be happy if a real user got this audit? Every fail got one plain line naming the first thing wrong. No categories yet.

An AI, acting as a second reviewer, judged the same 41 independently. We agreed on 28, and I made the final call on the rest. Each of us caught a different kind of mistake, which is the point of a second opinion: it shows where your own standard is vague.

What the notes said

Grouped like an affinity map, the notes fell into six kinds of failure. Two tied for biggest, at 7 failed audits each. One was the tool reporting things a screenshot cannot show as if it had seen them: hover states, what sits below the scroll, whether undo exists, what happens on click. A screenshot is one frame, so advice about the rest is a guess. The other was asking for confidence scores on ordinary chat replies, where nothing was being researched and certainty would not change what the user did next.

The first became the written standard: every piece of advice is either seen (it points at something on the screen) or assumed (it depends on something the screen cannot show). Of 116 pieces of advice, 93 were assumed.

Checking the AI reviewer before trusting it

Labelling 116 pieces of advice by hand every time the tool changes does not scale, so an AI reviewer was briefed to apply the seen-or-assumed rule. Before trusting it, its verdicts were compared with the human ones. Half the advice was used to tune its brief; the other half was held back and scored once, with no edits.

On the tuning half it caught 45 of 50 problems. On the held-back half, the honest number, it caught 41 of 44 and raised 5 false alarms in 11 clean cases. Good enough to count with, as long as someone still skims what it flags.

Two fixes, measured

The first fix told the tool to stop guessing. Assumed advice went from 93 to 86, and two audits broke outright. The second fix gave the guesses somewhere to go: a separate Worth checking list of questions for the designer to answer, kept apart from the gaps the tool can actually see. Assumed advice went from 93 to 73, and honest gaps grew from 23 to 34.

Then the fix was tested on 19 screenshots it had never seen. The drop was smaller there: from 92% assumed to 88%. Part of the improvement only suited the examples it had been tuned on. Without the fresh set, that would have shipped as a bigger win than it was.

Smaller findings

A vague rule, made exact

The tool kept asking ordinary chat replies for confidence scores. Narrowing the rule to "only where the AI outputs a score, prediction or classification" took those wrong flags from 8 to 0.

A fix that hurt

A rule to spot marketing images and skip them also skipped real product screens. It was reverted. Without re-running the full set, it would have looked like a fix.

Broken output

Some audits failed because the model wrote invalid JSON. That was fixed in the code that reads it, and the final run had 0 broken audits.

Still wrong

After both fixes, a human review passed 33 of the 41 audits and failed 8. One site, Our World in Data, got worse and is logged to fix next.

What I learned

By running evals I knew where my product stands, where it fails and where it needs improvement. Those are things I only knew after conducting the evals.

My advice to other designers: learn how to conduct evals and understand the topic, as part of your research and of thinking through your approach to the product's design.