How to build an eval set from production traffic
Sample real traces with a written mix (random, clustered, flagged), have your domain expert label them pass or fail with reasons, add every past bug as a case, remove or mask personal data, and store the set as versioned JSONL next to your code. Refresh it on a schedule, so it keeps matching what users ask. Your best eval cases are your worst production moments, each turned into a test with a reason attached.
Why real traffic beats an imagined test set
Eval sets written before launch test what the team imagined users would ask. Real users ask differently: shorter, vaguer, in mixed languages, with typos, with context the system doesn't have, and sometimes trying to break it. A set built from production tests the system you have against the users you have.
Before launch, you have no production traffic, so start with hand-written cases and the people closest to users (support, sales, domain experts), and replace them with real traces as soon as they exist.
Step 1: sample with a written mix
A starting point:
| Source | Share |
|---|---|
| Random production sample | 40% |
| Clustered by topic | 30% |
| Flagged failures (feedback, retries, judge flags) | 20% |
| Hand-written edge cases and red-team cases | 10% |
See how to pick which traces to read for each sampler.
Step 2: label with your expert
Your domain expert labels each trace pass or fail, with a short reason, and tags it with a failure mode from your error analysis. Where possible, also record what a correct answer must contain, so the case can be checked with code.
Step 3: add every past bug
Every bug that reached users becomes a case, with the date and a one-line reason. "The date math bug from March" is the kind of case that makes a regression suite valuable.
Step 4: remove personal data
Production traces contain names, emails, phone numbers, order numbers and sometimes health or financial details. Mask or replace them before the trace enters the eval set, and keep the mapping out of the repository. Check that the masked version still triggers the same behaviour, because masking can change what the system does. See data leakage and PII.
Step 5: store and version it
Store one case per line as JSONL, versioned with your code.
import json
from datetime import date
case = {
"id": "refund-eu-0042",
"input": "I'm in [COUNTRY:DE], bought headphones 10 days ago, can I return them?",
"context_ids": ["policy-refunds-2025-eu"],
"check": {"must_include": ["14 days"], "must_not_include": ["30 days"]},
"failure_mode": "wrong_region_policy",
"reason": "Quoted the US 30-day window to an EU customer.",
"source_trace": "tr_8f3a91",
"added": str(date(2026, 9, 14)),
"label": "fail",
}
line = json.dumps(case, ensure_ascii=False)
print(line[:120] + "...")
print(sorted(json.loads(line)))
Each record carries the input, any context it needs, the check or expected behaviour, the failure mode, the reason, the source trace ID and the date it was added.
Step 6: keep it fresh
Your users, product and documents change, so the eval set ages. Track two numbers:
- Age: the share of cases added in the last 90 days.
- Coverage: the share of known failure modes with at least a handful of cases.
Add new cases every week from monitoring, and review old ones in your quarterly eval review.
A first version in a week
- Export 200 traces with the written mix.
- Have your expert label 100 of them, with reasons.
- Add your known past bugs.
- Mask personal data.
- Commit version 1 of the eval set, and run it.
From the floor
In the RAG project lab, the test cases we wrote for CI checked the chunks and the pipeline with data we prepared. Once a system like that is live, the step this page describes replaces most of those prepared cases with real questions from users, labelled by someone who knows the domain.