How to check if your LLM judge is right
An LLM judge is a model that grades another model's output, and it makes mistakes like any model. To check it, run it on traces your expert has already labelled and measure two numbers separately: how often it agrees with the expert on passes (true positive rate) and on failures (true negative rate). Plain accuracy hides a judge that misses failures, and once you know both rates, you can correct the pass rate the judge reports.
Why judges need checking
A judge that is wrong 20% of the time can happily tell you your product passes 95% of the time. If you trust that number, you ship regressions with a green dashboard.
LLM judges are useful because they can check things code can't, like whether an answer follows a policy or whether a tone is appropriate. That's exactly why they need to be validated against a person who knows what right looks like.
How to check a judge
- Take traces your expert has already labelled pass or fail, with reasons.
- Run the judge on the same traces.
- Compare the judge's labels with the expert's in a 2 by 2 table.
| Expert: pass | Expert: fail | |
|---|---|---|
| Judge: pass | Agreed pass | Judge too lenient |
| Judge: fail | Judge too strict | Agreed fail |
Why accuracy misleads
If 90% of your traces pass, a judge that always says "pass" scores 90% accuracy and catches zero failures. The failures are the reason the judge exists.
So measure two numbers, with pass as the positive class:
- True positive rate (TPR): of the outputs the expert passed, what share did the judge pass?
- True negative rate (TNR): of the outputs the expert failed, what share did the judge fail?
The evals FAQ gives an example: if people find 10 failures and the judge catches 8, that rate is 80%, and if people find 100 good outputs and the judge passes 95, that rate is 95%.
Judges are often too lenient: they pass things a strict expert would fail. A low TNR is the most common problem, and the most dangerous.
Correcting the reported pass rate
Once you know the judge's error rates, you can estimate the true pass rate from the rate the judge reports.
The open-source judgy library from Husain and Shankar's course uses this formula, with pass as the positive class:
corrected = (observed + TNR − 1) / (TPR + TNR − 1)
def rates(expert: list[str], judge: list[str]) -> tuple[float, float]:
pairs = list(zip(expert, judge))
passes = [j for e, j in pairs if e == "pass"]
fails = [j for e, j in pairs if e == "fail"]
tpr = passes.count("pass") / len(passes)
tnr = fails.count("fail") / len(fails)
return tpr, tnr
def corrected_pass_rate(observed: float, tpr: float, tnr: float) -> float:
if tpr + tnr <= 1:
raise ValueError("judge is no better than chance")
return min(1.0, max(0.0, (observed + tnr - 1) / (tpr + tnr - 1)))
expert = ["pass"] * 20 + ["fail"] * 10
judge = ["pass"] * 19 + ["fail"] * 1 + ["pass"] * 3 + ["fail"] * 7
tpr, tnr = rates(expert, judge)
print(f"TPR {tpr:.2f} TNR {tnr:.2f}")
print(f"judge reports 80% pass -> corrected {corrected_pass_rate(0.80, tpr, tnr):.0%}")
With TPR 0.95 and TNR 0.70, a reported pass rate of 80% corrects to about 77%. This only works when TPR + TNR is above 1, which means the judge does better than a coin flip.
When judge and expert disagree
Read the disagreements before changing anything. Usually the fix is a clearer rubric or a better example in the judge prompt, not a bigger model. Common causes:
- The rubric is vague, so the judge fills the gap with its own idea of quality.
- The judge doesn't see information the expert used, like the policy document.
- The judge is asked to check several things at once. Split it into one judge per failure mode.
When you change the judge prompt, avoid tuning it on the same examples you measure it on. See how to stop a judge prompt from overfitting.
Criteria change as you grade
Shreya Shankar and her UC Berkeley colleagues studied how people build LLM judges (UIST 2024). They found that "users need criteria to grade outputs, but grading outputs helps users define criteria". The judge and the human standard change together, so keep re-checking the judge against fresh expert labels, not only once at the start.
From the floor
In the ML system design session, I used a cancer prediction example to explain why accuracy alone can mislead: a model can show 99% accuracy, and if the 1% it gets wrong are patients who had cancer and were told they didn't, it is a bad model. The same thing happens with a judge that rarely says fail: high agreement overall, and the failures you care about slip through.