The interface is built. The question that matters after launch is whether the answers inside it are any good. This lesson introduces evals, the method teams use to check AI answers against a written standard, and points to a full course on running one.
Why a chat interface needs more than a design review
Every lesson so far designs the container: the bubbles, the streaming, the errors, the controls. What appears inside it is generated fresh every time. Ask the same question twice and the wording, the order and sometimes the conclusion can change. Ask something nobody planned for and the AI will still answer confidently.
A design review checks a screen once. The answers in a chat need checking again and again, against a standard someone has written down. That repeatable check is an eval.
What an eval is
An eval has two parts: a written standard for what a good answer is, in plain words, and a repeatable way to check real answers against it. With both in place you can count how often the chat meets the standard, and see whether a change to the prompt or the model made it better or worse.
Collect real conversations
- Start with 30 to 50 real or realistic chats, including awkward ones: off-topic requests, frustrated users, questions the product cannot answer.
Read each answer and mark it
- Pass or fail, plus one line in plain words on the first thing that is wrong. No categories yet.
Group the notes and count
- Cluster the notes the way you would affinity-map research findings. The biggest group is the first thing to fix, and the first rule to write down.
Several earlier lessons turn into yes or no rules an eval can test on every release: the AI says what it cannot do and offers an alternative (Lesson 7), voice replies stay to two or three sentences (Lesson 8), and the AI asks before any irreversible action (Lesson 11).
An eval of an AI audit tool found it kept asking for confidence scores on ordinary chat replies, where nothing was being researched and certainty did not change what the user did next. Once that was written down as a rule and the tool was told when confidence actually matters, the wrong flags went from 8 to 0. Not every chat reply needs a confidence indicator; an eval is how you find out which ones do.