LLM hallucination: causes and the controls that work
A hallucination is when a model states something false or unsupported as if it were true: a made-up fact, a citation that doesn't exist, or an answer that contradicts the document it was given. It happens because the model generates the most plausible text, not verified text, and because training and evaluation have rewarded confident guessing over saying "I don't know". You can't remove it completely, but you can reduce it a lot and catch most of what remains: ground answers in retrieved sources, verify citations in code, allow and reward abstention, use tools for exact values, validate outputs, and measure the hallucination rate on your own test set.
What counts as a hallucination
The word covers several different failures, and they have different fixes.
- Made-up facts: a wrong date, a person who doesn't exist, a feature your product doesn't have.
- Fabricated sources: citations, case law, papers, URLs or quotes that look real and aren't.
- Unfaithful answers: the right documents were in the context, and the answer says something they don't say.
- Wrong calculations or reasoning presented confidently.
- Misread tool results: the tool returned the right data and the model reported it wrongly.
It helps to separate two questions when you investigate one. Was the answer unsupported by the information the model had (a faithfulness problem), or was the information it had wrong or missing (a retrieval or data problem)?
Hallucination is not nondeterminism
Hallucination means a wrong answer. Nondeterminism means different answers to the same input. A model can be perfectly consistent and consistently wrong, so fixing one doesn't fix the other.
Why models hallucinate
They generate plausible text, not true text
An LLM is trained to predict the next token. Nothing in that objective checks whether a statement is true. For things that follow strong patterns, like grammar, common facts and well-known code, the most plausible continuation is usually correct. For rare facts, like a specific person's birthday, a minor court case or your company's internal policy, there is no pattern to follow, and the model produces something that sounds right.
Training and evaluation reward guessing
In September 2025, researchers at OpenAI published an explanation that changed how many people think about this. Most benchmarks score only accuracy, like a multiple-choice exam with no penalty for wrong answers. If a model doesn't know someone's birthday and guesses a date, it has about a 1 in 365 chance of being right, while "I don't know" always scores zero. Over thousands of questions, the guessing model looks better on the leaderboard, so training that optimises for those scores teaches models to guess confidently. Their proposed fix is to change evaluations so confident errors are penalised more than honest uncertainty.
The lesson for your application is the same: if your prompts, your evaluations and your product only reward answers, you will get guesses.
Missing information
Everything after the model's knowledge cutoff, everything private to your company, and anything the retrieval step failed to find is unknown to the model. Asked about it anyway, it often fills the gap instead of saying so.
Prompts that invite it
A question that assumes something false ("Why did our refund policy change in March?") pushes the model to explain a change that never happened. Long, cluttered contexts, conflicting documents and vague questions all raise the rate too.
What it costs when it goes wrong
In 2023, lawyers in a US federal case, Mata v. Avianca, submitted a brief that cited court decisions generated by ChatGPT. The cases didn't exist. The court fined the lawyers and their firm $5000, and the case became the standard example of why legal teams now insist on verified citations.
In most applications, the cost is quieter: a support bot that promises a refund the policy doesn't allow, a summary that adds a number that wasn't in the report, an agent that acts on a value it misread. Each one costs trust, and sometimes money.
Controls that work
No single control is enough, and the right mix depends on how costly a wrong answer is. These are ordered roughly from the first thing to do to the last.
Ground answers in sources
Give the model the information it needs in the prompt, usually with retrieval-augmented generation, and instruct it to answer only from that information. Grounding turns "remember the answer" into "read the answer", which models do much more reliably. It only works if retrieval finds the right documents, so measure retrieval quality separately.
Make it cite, and verify the citations in code
Ask the model to cite the source of each claim, using IDs of the chunks you provided, not free-form references. Then check in code that every cited ID was actually provided, and, for important claims, that the cited text supports the claim, with a second model call or a smaller checking model. A citation the model invented, or one that doesn't support the sentence, is caught before the user sees it.
Allow and reward "I don't know"
Tell the model explicitly that it should say when the sources don't contain the answer, and give it a clear format for that, like a field "answerable": false.
Then make sure your evaluation counts a correct "I don't know" as a success, and a confident wrong answer as worse than no answer.
Add unanswerable questions to your test set on purpose, so you can see whether the model abstains.
Use tools for exact values
Don't ask the model to recall or compute values that must be exact: prices, balances, dates, stock levels, calculations, order statuses. Give it tools that look them up or compute them, and have the answer use the tool result.
Constrain and validate the output
Ask for structured output and validate it. Check values against sources of truth where you can: a product mentioned must exist in your catalogue, a total must equal the sum of the line items, a date must be in a valid range.
Add a verification step for high-stakes answers
For important outputs, a second step can check the first: another model call that compares the answer with the sources and flags unsupported claims, or a smaller model trained for that check. Some systems put filters on both sides of the main model, one checking the input, for example for prompt injection or policy, and one checking the output before it reaches the user.
Narrow the scope
A model asked to do one specific job, with clear boundaries, hallucinates less than a model asked to be a general assistant. Tell it what it should and shouldn't answer, and route out-of-scope questions somewhere else.
Keep a person in the loop where it matters
For legal, medical, financial and other high-stakes decisions, the model drafts and a qualified person approves. Design the interface so the person can see the sources and check them quickly, otherwise review becomes a rubber stamp.
What doesn't work on its own
- Telling the model "don't hallucinate" or "only say true things". It doesn't know which of its statements are false.
- Setting temperature to 0. It makes outputs more consistent, not more correct.
- Using a bigger or newer model and assuming the problem is solved. Better models hallucinate less on average, and still do it on rare facts, private data and questions with false assumptions.
- A reasoning model. Thinking longer helps with logic, not with facts the model doesn't have.
Measure it
You can't manage what you don't measure. Build a test set from real questions that includes:
- Questions the sources answer clearly.
- Questions the sources answer only partly.
- Questions the sources don't answer at all.
- Questions with false assumptions.
For each answer, check whether every claim is supported by the provided sources, whether citations are real and relevant, and whether the model abstained when it should have. Track the hallucination rate and the abstention rate over time, and run the set whenever you change the model, the prompt or the retrieval. Evals and reliability covers how.
In production, give users an easy way to report a wrong answer, and review those reports, because they show you the failures your test set missed.
From the floor
A legal technology company working with individual lawyers reached out to me, and before proposing anything, I researched the problems in their market, using deep research tools to map the industry and then asking which problems they were unlikely to solve themselves. One of the strongest candidates was a hallucination citation verifier for large law firms: a service that checks every case citation in an AI-generated draft against legal databases, because a fabricated citation is one of the biggest liability risks for a firm using AI. We then built a version of it in the cohort as a project.
Through that research, I kept a rule I give builders too: research reports from AI tools are a starting point, not a source of truth, because they can contain the same kind of confident mistakes, so every number and claim in them gets checked before it goes into a proposal.
The architecture we discussed for it puts filters on both sides of the LLM: a small, fast model checks the input for prompt injection and compliance issues before the expensive LLM call, and another check runs on the output before anything reaches the user.
Related
- Evals and reliability
- Retrieval and RAG
- Structured outputs and function calling
- Security and governance