When to retire an LLM eval
Evals go stale in two ways: they get too easy and stop telling you anything, or your idea of good changes and they keep checking the old idea. Before retiring a check that always passes, ask what it protects: a check for a serious failure stays in the regression suite, and a check for a minor issue that always passes is only costing CI time. Version every rubric change, re-check judges when the rubric changes, and put an eval review on the calendar, maybe once a quarter.
Two kinds of staleness
Saturation
The evals FAQ is direct: if an eval keeps passing every time, run it less often or retire it. Anthropic calls this saturation: the agent passes every task it can solve, so the score has no room left to move, and the eval no longer tells you anything new.
Criteria drift
Your idea of good changes as you look at more outputs. Shreya Shankar's 2024 study showed that grading outputs changes what people think the rules should be. That's healthy, as long as you write it down. An eval still checking last year's definition of good is measuring the wrong thing with full confidence.
What to do with an eval that always passes
Ask what it protects.
- A check for a serious failure that always passes is a regression guard. Keep it in the regression suite, because the day it fails is the day it earns its place.
- A check for a minor issue that always passes is just costing CI time and money. Retire it, or run it weekly instead of on every pull request.
Versioning rubrics
Give every rubric change a version number, a date and a one-line reason.
handoff_rubric
v1 2026-03-02 Fail if the bot doesn't offer a human when asked.
v2 2026-05-18 Also fail if it offers a human for questions it can answer (support load).
v3 2026-09-07 Exclude after-hours: offer a callback instead of live chat.
When the rubric changes, re-check any LLM judge built on the old version against fresh expert labels. A judge validated against v1 isn't validated against v3.
Broken evals
Watch for evals that are broken rather than hard. If a top model scores 0% on a task across many tries, the task or the grader is usually broken. Read the failing cases before deciding the system is bad.
Quarterly review
Put an eval review on the calendar, maybe once a quarter. For each check, look at:
- Its pass rate history.
- Its cost per run.
- The last time it failed, and whether that failure was real.
- Whether its rubric still matches what the business wants.
Then decide: keep, rewrite, run less often, or retire.
| Check | Pass rate (90 days) | Last real failure | Cost per run | Decision |
|---|---|---|---|---|
| PII in output | 100% | 2026-02 | Low | Keep (regression guard) |
| Greeting uses name | 100% | Never | Low | Retire |
| Refund window by region | 94% | Last week | Medium | Keep |
| Tone rubric v1 | 71% | Rubric changed | High | Rewrite to v3, re-check judge |
A real example
OpenAI built SWE-bench Verified in August 2024 to fix problems in the original SWE-bench. Eighteen months later, its own audit found 59.4% of 138 hard tasks had real problems, and models could reproduce some fixes from memory. Scores had crept from 74.9% to 80.9%, and nobody could tell how much was real skill, so OpenAI retired it. If a team that size retires its flagship benchmark, your golden set from last year deserves a review too.
From the floor
I haven't covered retiring evals in a session yet, so this play comes from the playbook research.
Related
- CI gates and production monitoring
- Benchmarks vs product evals
- Check your LLM judge
- Evaluate coding agents