How to stop an LLM judge prompt from overfitting
Every time you edit a judge prompt and re-test it on the same examples, you tune the prompt to those examples, which is overfitting, the same problem as in classic machine learning. Split your expert labels into three sets: about 10 to 20% for train (the only examples you may copy into the prompt), 40 to 45% for dev (test after every edit) and 40 to 45% for test (look once, at the end). Make sure dev and test each have 30 to 50 passing and 30 to 50 failing examples, or you can't measure whether the judge catches failures.
How a judge prompt overfits
You write a judge prompt, run it on 40 labelled traces, and it disagrees with the expert on 6. You read those 6, adjust the wording, add one of them as an example, and run again. Now it disagrees on 2.
Has the judge improved, or has it learned those 40 traces? You can't tell, because you measured it on the same data you used to fix it. Repeat this ten times and you'll have a judge that looks excellent on your examples and behaves differently on new traffic.
Train, dev and test
Collect expert labels for each failure mode you want a judge for. Husain and Shankar suggest 100 to 200 labelled examples per failure mode.
| Split | Share | Used for |
|---|---|---|
| Train | 10 to 20% | The only examples you can put in the judge prompt as few-shot examples |
| Dev | 40 to 45% | Testing the judge after every prompt edit |
| Test | 40 to 45% | Reporting final TPR and TNR, once |
It's the same idea as the train, validation and test split for fine-tuning data: the data you learn from, the data you tune on, and the data you only use to report.
Balance passes and failures
Make sure dev and test each have 30 to 50 passing examples and 30 to 50 failing ones. A dev set with 4 failures can't tell you whether the judge catches failures.
If failures are rare in your traffic, a natural sample might give you almost none. Go and find more with targeted sampling (play 5): flagged traces, thumbs down, outliers.
import random
def split_labels(rows: list[dict], seed: int = 42) -> dict[str, list[dict]]:
rng = random.Random(seed)
out = {"train": [], "dev": [], "test": []}
for label in ("pass", "fail"):
group = [r for r in rows if r["label"] == label]
rng.shuffle(group)
n = len(group)
a, b = int(n * 0.15), int(n * 0.575)
out["train"] += group[:a]
out["dev"] += group[a:b]
out["test"] += group[b:]
return out
rows = [{"id": i, "label": "pass" if i % 2 else "fail"} for i in range(160)]
splits = split_labels(rows)
for name, part in splits.items():
fails = sum(r["label"] == "fail" for r in part)
print(f"{name:<5} {len(part):>3} examples, {fails} failures")
Splitting passes and failures separately (stratifying) keeps both sets balanced.
Rules that keep the judge honest
- Never put a test example into the judge prompt, not even rewritten. A judge that has seen the answers looks great in your report and bad in production.
- Change one thing at a time while tuning: the rubric wording, one example, or the output format. Write down which change moved TPR or TNR.
- Look at the test set once, at the end, to report the judge's final rates. If you tune after looking, it has become a dev set, and you need a fresh test set.
- Rebuild the split when your product or rules change. A judge checked on last quarter's traffic is only checked for last quarter's traffic.
A real example of leakage
OpenAI's February 2026 audit of SWE-bench Verified found that frontier models could reproduce some of the benchmark's original fixes from training, so high scores reflected exposure as much as skill. A judge prompt leaks the same way when test examples slip into its few-shot block.
From the floor
In the fine-tuning module, every dataset we prepared was split into training, validation and test sets with a fixed seed, and the rule was that the model is evaluated on data it never trained on. A judge prompt needs the same discipline, because its few-shot examples are its training data.