Why pass or fail beats a 1 to 5 score for LLM evals
A 1 to 5 score feels more detailed than pass or fail, but nobody can define the difference between a 3 and a 4 consistently, and people drift to the middle when unsure, so failures hide in a pile of 3s. Pass or fail forces you to define exactly what failure means, which is most of the value of an eval. If you want more detail, split one fuzzy question into several yes or no checks, each tied to one failure mode.
What goes wrong with a 1 to 5 scale
A Likert scale asks someone to rate an output from 1 to 5. Ask three reviewers what separates a 3 from a 4 and you'll get three answers. Ask the same reviewer next week and you may get a fourth.
Husain and Shankar note that annotators often default to middle values to avoid hard decisions. So the outputs that should have failed sit at 3, next to the ones that were fine, and your average looks stable while users complain.
Why pass or fail works better
It forces a definition
To label something pass or fail, you have to say what failure means. "Fails if it quotes a refund window that doesn't match the customer's region" is a definition someone else can apply. That definition is most of the value of the eval.
It compares cleanly with an LLM judge
With binary labels, you can measure directly how often a judge agrees with your expert on passes and on failures. See how to check if your LLM judge is right.
It produces numbers a team can act on
"74% pass rate on handing off to a human when asked" is a number that moves when you ship a fix, and everyone understands it. "Average quality 3.6" isn't.
Each check points to one fix
If every check targets one failure mode from your error analysis, every failing check tells you what to fix.
Getting detail without a scale
Split one fuzzy question into several yes or no checks. "Is the answer complete?" becomes:
- Did it include the price?
- Did it include the deadline?
- Did it include the cancellation fee?
- Did it link the policy page?
Then you can report something clear, like "4 out of 5 expected facts included" or "price included in 97% of answers".
def check_answer(answer: str) -> dict[str, bool]:
text = answer.lower()
return {
"has_price": "₹" in answer or "$" in answer,
"has_deadline": "days" in text,
"has_fee": "fee" in text,
"links_policy": "/refund-policy" in text,
}
answer = "You can cancel within 14 days for a ₹499 fee. See /refund-policy for details."
results = check_answer(answer)
print(results)
print(f"{sum(results.values())} of {len(results)} checks passed")
A pass rate of 100% isn't the goal
A suite where everything passes isn't a win. It usually means the checks aren't testing anything hard. Husain and Shankar note that a pass rate around 70% may mean your evals are testing something meaningful. Add checks for the failures you haven't solved yet, and track their pass rates going up.
Where scores still make sense
Some things are truly numeric: latency, cost, retrieval recall, number of steps. Measure them as numbers, then set a threshold and gate on pass or fail: "p95 latency under 3 seconds", "recall@5 at least 0.85".
A real example
Honeycomb's natural language query assistant aligned its LLM judge with its domain expert using pass or fail labels plus written reasons, and reached over 90% agreement in three rounds.
From the floor
In the ML system design session, the evaluation example I used was a harmful content detector: what matters is whether harmful content was detected, because a post that was harmful and passed as not harmful is a failure, whatever the overall score says. That is a pass or fail definition tied to the failure that matters.