Eval-driven development for LLM applications
Eval-driven development means every change to an LLM system starts with a test that captures the behaviour you want, fails on the current system, and passes after your change, without breaking the existing tests. It's test-driven development adapted to outputs that vary from run to run, so you run each case several times and check rules instead of exact strings. The loop is cheap once the first 30 cases exist, and the expensive part is starting, which is why the first eval set should come from traces you already have.
Why change by feel doesn't work
Without evals, prompt and model changes are judged by trying a few examples by hand. The change fixes the example you were looking at, and quietly breaks two others you weren't. Every fix is a guess, and regressions reach users before anyone notices.
In normal software, test-driven development solves this: write a failing test, make it pass, keep all other tests passing. LLM systems need the same discipline with two adjustments: the output varies, and correctness is often a rule rather than an exact value.
How the loop works
- Find a failure, in traces, user reports or your own error analysis.
- Write the smallest test case that reproduces it, with a pass or fail check.
- Run it several times on the current system and confirm it fails. If it passes sometimes, record how often.
- Change one thing: the prompt, retrieval, a tool, the model.
- Run the new case and the full regression suite.
- Merge only if the new case passes reliably and nothing else regressed.
A minimal eval harness
import statistics
def run_case(system, case: dict, trials: int = 5) -> float:
passes = [case["check"](system(case["input"])) for _ in range(trials)]
return statistics.mean(passes)
def run_suite(system, cases: list[dict], min_rate: float = 0.8) -> bool:
ok = True
for case in cases:
rate = run_case(system, case)
status = "PASS" if rate >= min_rate else "FAIL"
ok &= rate >= min_rate
print(f"{status} {rate:.0%} {case['name']}")
return ok
def current_system(question: str) -> str:
return "Refunds are available within 30 days."
cases = [
{
"name": "eu_refund_window",
"input": "I'm in Germany, how long do I have to return this?",
"check": lambda out: "14 days" in out,
},
{
"name": "mentions_refund",
"input": "Can I get my money back?",
"check": lambda out: "refund" in out.lower(),
},
]
print("suite passed:", run_suite(current_system, cases))
The EU case fails on the current system, which is exactly where the loop starts. After the fix, it should pass on every trial, and the other case should keep passing.
How to organise it in a project
- A
tests/evals/folder with one file per failure mode, each with its cases and checks. - A script that runs everything and prints pass rates per failure mode.
- The same script in CI, with the regression suite as a gate (play 19).
- Prompts versioned in the repository, so every eval result is tied to a prompt version.
What counts as one change
Change one thing at a time, and write down which change moved which pass rate. If you change the prompt, the retrieval settings and the model together, a better result tells you nothing about which change helped, and a worse one tells you nothing about which to roll back.
Measuring the habit
A useful number for the team: the share of production incidents that had a test written for them within a week. When that number is high, the same failure can't reach users twice.
From the floor
When we built the sentiment classification pipeline with a local Llama model, the files were a data processing step, the classifier, an evaluation file with accuracy, F1 and a classification report, a visualisation step and a pipeline that runs them all, and the rule I gave was to work on evaluation in parallel with the model, because then you can see while building whether the approach is going right and change the classifier or the processing before it's too late.
Related
- Find failure modes in traces
- Prompt testing
- CI gates and production monitoring
- Eval set from production