Guide
How to build your first AI evaluation set
A step-by-step guide to collecting, labelling and using a test set that tells you whether an AI system is good enough, and whether a change made it better.
2 min readLast reviewed
An evaluation set is a collection of realistic inputs with a definition of what a good output looks like. It is the single most useful asset in an AI project: it turns opinions into measurements, makes model upgrades safe and gives risk teams evidence. This guide walks through building a first version in a few weeks.
Step 1: collect real inputs
Pull genuine examples from where the work happens: tickets, emails, questions, documents. Aim for 100 to 300 items to start. Include common cases in proportion and deliberately add hard cases, edge cases and inputs where the correct behaviour is to refuse or escalate.
Step 2: define good
For each item, record what a correct result looks like. Depending on the task this may be an exact label, a set of required facts, the source document that should be cited, or a rubric for judging free text. Domain experts must own these definitions.
- Classification: the correct label, and acceptable alternatives if any.
- Extraction: the correct value for each field, or “absent”.
- Question answering: the facts the answer must contain and the source passages that support them.
- Agent tasks: the expected end state and any actions that must not happen.
Step 3: choose scoring methods
| Method | Use for | Watch out for |
|---|---|---|
| Exact or rule-based checks | Labels, fields, formats, forbidden actions | Brittle on free text |
| Model-graded scoring | Faithfulness, relevance, rubric criteria | Needs calibration against humans |
| Human review | Ground truth, nuanced quality | Slow and costly; sample it |
Step 4: calibrate automated judges
If you use a model to grade outputs, have two people grade a sample of 50 to 100 items as well. Measure agreement between the people, and between people and the model. Only rely on the automated grader for criteria where its agreement with humans is close to the agreement between humans.
Step 5: run it on every change
Automate the run in your delivery pipeline so that changes to prompts, retrieval, tools or models produce a score report. Compare against the previous version item by item, not only on averages, because improvements in one area often hide regressions in another.
Step 6: keep it alive
Add production failures and new types of input as they appear. Retire items that no longer reflect reality. Version the set alongside the code. Store it with the same care as production data, because it often contains real customer or employee content.