RAG failure modes: a catalogue with fixes
RAG systems fail in predictable places: documents parsed badly, answers split across chunks, the right chunk not retrieved or ranked too low, outdated or conflicting documents, permission leaks, and a model that ignores, mixes or goes beyond the context. For any wrong answer, first check whether the right passage was retrieved, because that one question tells you whether to fix retrieval or generation. This page lists the common failures with how to recognise them and what usually fixes them.
How to investigate a wrong answer
- Look at the exact chunks that were sent to the model.
- Check whether the correct information was among them.
- If not, check whether it exists in the index at all, and how it was parsed and chunked.
- If it was there, check how the model used it: ignored, misread, mixed with another chunk, or contradicted.
- Add the question to your evaluation set before fixing it, so you can confirm the fix and catch it if it returns.
Ingestion failures
Bad parsing
Symptom: answers about tables are wrong or vague, scanned documents never appear, text from different columns is mixed. Fix: use a layout-aware parser, OCR or a vision model for scans, and read a sample of parsed output. See document parsing.
Missing documents
Symptom: the system says it doesn't know about something that is definitely documented. Fix: check ingestion logs for failed files, unsupported formats, files too large, or sources not connected.
Stale or duplicate content
Symptom: answers quote an old version of a policy, or different answers to the same question on different days. Fix: track document versions, delete old chunks when a document is updated, deduplicate near-identical documents, and store dates to filter or rank by freshness.
Chunking failures
Answer split across chunks
Symptom: answers are incomplete, like the condition without the exception, or the number without what it refers to. Fix: structure-aware chunking, larger chunks or overlap, or small-to-big retrieval that sends the parent section. See chunking strategies.
Chunks without context
Symptom: the right chunk is retrieved but the model can't tell what it refers to ("the limit is 30 days" without saying which limit). Fix: add document titles and section headings to chunks, or contextual chunking.
Chunks too large
Symptom: retrieval is imprecise, and the prompt fills with irrelevant text. Fix: smaller chunks, tested on your evaluation set.
Retrieval failures
Vocabulary mismatch
Symptom: questions phrased differently from the documents retrieve nothing useful. Fix: hybrid search, query rewriting or expansion, or a better embedding model.
Exact terms missed
Symptom: questions with product codes, error codes or names retrieve related but wrong chunks. Fix: hybrid search with BM25.
Right chunk ranked too low
Symptom: the correct chunk is in the top 30 but not in the top 5 you send. Fix: reranking.
Similar but wrong entity
Symptom: answers about the X300 when the user asked about the X200, or about the wrong company. Fix: metadata filters, entity extraction from the question, hybrid search, reranking.
Follow-up questions fail
Symptom: the first question works, "and for laptops?" doesn't. Fix: rewrite follow-ups into standalone questions. See query rewriting.
Multi-hop and whole-corpus questions
Symptom: questions that need several documents or the whole collection get partial or shallow answers. Fix: decomposition, agentic RAG, or GraphRAG for relationships and themes.
Over-filtering
Symptom: no results for questions that should have answers. Fix: check filters, add a fallback to broader search, and tell the model when results were limited.
Security failures
Permission leaks
Symptom: users get answers from documents they can't open. Fix: enforce permissions as filters in the retrieval query from the user's identity. Never rely on the prompt. See metadata filtering and access control.
Prompt injection through documents
Symptom: a document or web page contains text like "ignore your instructions and…", and the model follows it. Fix: treat retrieved text as data, delimit it clearly, limit what tools the model can call, and validate outputs. See security and governance.
Generation failures
Answer not grounded
Symptom: the answer contains claims that aren't in any retrieved chunk. Fix: instruct the model to answer only from the sources, require citations, verify citations in code, and measure faithfulness.
Guessing instead of abstaining
Symptom: confident answers to questions the documents don't cover. Fix: tell the model exactly what to say when the sources don't contain the answer, test with unanswerable questions, and count correct abstentions as passes.
Conflicting sources
Symptom: the answer mixes an old and a new policy, or picks one without saying so. Fix: filter to current versions, include dates in chunks, and instruct the model to prefer the newest source and mention conflicts.
Ignoring the middle
Symptom: the answer uses the first and last chunks and ignores the one in the middle that had the answer. Fix: send fewer chunks, put the most relevant first, rerank.
Wrong numbers
Symptom: amounts, dates or counts in answers don't match the documents. Fix: for important values, look them up from structured data with a tool and use the exact value. See hallucination.
Fabricated citations
Symptom: the answer cites a source ID that wasn't provided, or a real one that doesn't support the claim. Fix: give each chunk an ID, check in code that every cited ID was provided, and verify support for high-stakes answers.
Operational failures
- Slow answers: too many chunks, slow reranking or a large model. Measure each stage and trim.
- Rising cost: more chunks per query, longer prompts, agent loops. Track cost per question.
- Quality drift: documents, users and models change. Run your evaluation regularly, not only at launch.
From the floor
In our advanced RAG session, the first failure we looked at was the "similar but wrong entity" problem: a question about Tesla brought back chunks about Apple, and a question about self-driving cars brought both, because a basic pipeline returns whatever is related without looking at what the user wants. That one example covers half of this page, because the fixes are metadata, ranking and filtering.