How to test if an LLM knows when to say "I don't know"
Abstention means the system declines to answer, asks a clarifying question or hands off to a person, and it's the right move when the answer isn't in the information available. Scoring only accuracy rewards guessing, so test abstention with a set of roughly equal answerable and unanswerable questions, and score both sides: did it abstain when it should, and did it answer when it could. Report the error rate on attempted answers next to accuracy, because "22% right, 26% wrong" and "24% right, 75% wrong" are very different products.
Why accuracy rewards guessing
On a multiple-choice exam, a guess can score points and a blank never does. OpenAI's 2025 paper on why language models hallucinate argues that the same is true of how most models are evaluated: scoring only accuracy rewards a model for guessing instead of saying it's unsure.
Their SimpleQA example shows the trade-off:
| Model | Abstained | Right | Wrong |
|---|---|---|---|
| gpt-5-thinking-mini | 52% | 22% | 26% |
| o4-mini | 1% | 24% | 75% |
The model with slightly higher accuracy was wrong almost three times as often. If you only report accuracy, you'd choose the second one.
Abstention is still unsolved
Meta's AbstentionBench (2025) tested 20 frontier models on 20 datasets and found that abstention is still an unsolved problem. Reasoning fine-tuning made abstention 24% worse on average: models trained to reason harder were more likely to push through to an answer instead of saying they didn't know. So don't assume a newer or smarter model handles it for you.
Building the test set
Build a set with roughly equal numbers of answerable and unanswerable questions. Make the unanswerable ones realistic, because obvious nonsense is easy to refuse:
- Questions just outside your knowledge base ("What is the refund policy for gift cards?" when only physical products are covered).
- Questions with a false premise ("Why did you raise the price of the basic plan in March?" when you didn't).
- Questions missing details the user hasn't given yet ("Can I cancel?" without saying which order or plan).
Scoring both sides
| System answered | System abstained | |
|---|---|---|
| Answer available | Check correctness | Over-refusal (failure) |
| Answer not available | Hallucination (failure) | Correct abstention |
Score two checks:
- On unanswerable questions, did it abstain, ask for the missing detail or hand off?
- On answerable ones, did it answer instead of refusing? Too many refusals is a failure too, because a system that refuses everything is safe and useless.
def abstention_report(results: list[dict]) -> dict[str, float]:
ans = [r for r in results if r["answerable"]]
unans = [r for r in results if not r["answerable"]]
attempted = [r for r in results if not r["abstained"]]
wrong = [r for r in attempted if not r["correct"]]
return {
"right_answers": sum(r["correct"] for r in results) / len(results),
"abstained_when_unanswerable": sum(r["abstained"] for r in unans) / len(unans),
"over_refusal": sum(r["abstained"] for r in ans) / len(ans),
"error_rate_on_attempts": len(wrong) / len(attempted),
}
results = (
[{"answerable": True, "abstained": False, "correct": True}] * 40
+ [{"answerable": True, "abstained": False, "correct": False}] * 4
+ [{"answerable": True, "abstained": True, "correct": False}] * 6
+ [{"answerable": False, "abstained": True, "correct": False}] * 35
+ [{"answerable": False, "abstained": False, "correct": False}] * 15
)
for name, value in abstention_report(results).items():
print(f"{name:<30}{value:.0%}")
In this example, 40% of all questions got a right answer, the system abstained on 70% of the unanswerable ones and refused 12% of the answerable ones, and almost a third of the answers it did attempt were wrong, which is the number a user who trusts the answer cares about.
Making the system abstain
- Tell the model explicitly that "I don't know" or a clarifying question is a correct answer, and give it the exact wording to use.
- In RAG, check whether the retrieved context can answer the question before generating, and route to an abstention when it can't.
- Give it a handoff tool for cases that need a person.
- Show examples of correct abstention in the prompt.
- Track over-refusal at the same time, so the fix doesn't swing too far.
Most serious failure in high-stakes domains
In law, finance and health, treat a confident made-up fact as your most serious failure type, and weight your evals and your release decisions that way.
In June 2023, in Mata v. Avianca, Judge P. Kevin Castel fined a New York law firm $5,000 after a brief cited six fake court cases generated by ChatGPT. In October 2025, Deloitte agreed to refund the final payment on an A$440,000 report for Australia's Department of Employment and Workplace Relations after it was found to contain non-existent references, produced with help from Azure OpenAI GPT-4o.
From the floor
The reason we fine-tuned for the legal LLM project lab was exactly this: in law, accuracy matters, and there have been cases where lawyers submitted reports generated with ChatGPT that contained mistakes and were fined.
In a later session, I worked through a hallucination citation verifier for legal platforms, starting from the question "does the case say what you claim?": extract each citation from a brief, retrieve the full case from a legal database, check whether it supports the claim, and return a confidence score with an audit trail, using a small fine-tuned legal model for the check instead of an expensive large model on every citation.