How to evaluate a RAG system: retrieval metrics vs answer metrics
Evaluate a RAG system in two halves, because they fail for different reasons and are fixed in different places. For retrieval, measure whether the right chunks come back: recall at k, precision at k and MRR, against a test set of real questions with known answer passages. For generation, measure whether the answer is faithful to the retrieved chunks, answers the question, is correct, cites real sources, and says "I don't know" when the chunks don't contain the answer. Then track both every time you change chunking, embeddings, retrieval settings, prompts or models.
Why "the answers look good" isn't enough
A RAG system can produce fluent, confident answers that are wrong, partly supported, or supported by the wrong document. Reading ten answers tells you very little about the thousands you didn't read.
And when an answer is wrong, you need to know why. Was the right chunk never retrieved, or was it retrieved and ignored? The fix for the first is in parsing, chunking, embeddings or search. The fix for the second is in the prompt, the number of chunks or the model. Only evaluating the two halves separately tells you which.
Build a test set
- Collect 50 to 200 real questions from users, support tickets, search logs or subject experts. If you don't have users yet, write questions the way users would, not the way documents are written.
- For each, record the passage or passages that contain the answer (chunk IDs, or document and page).
- Write the expected answer, or the key facts it must contain.
- Include questions the documents can't answer, where the right response is "I don't have that information".
- Include hard cases: follow-up questions, exact identifiers, questions spanning two documents, questions with outdated information in some documents.
- Keep it in a versioned file, and add every production failure to it.
You can generate candidate questions with an LLM from your documents to get started faster, then review them by hand, because generated questions tend to copy the document's wording and are easier than real ones.
Retrieval metrics
These need only the test set and your retrieval system, no LLM.
- Recall at k: the share of questions where at least one correct passage appears in the top k results. This is usually the most important number, because the model can't use a passage it never receives.
- Precision at k: the share of the top k results that are relevant. Low precision means more noise in the prompt.
- MRR (mean reciprocal rank): how high the first correct passage ranks, averaged over questions. 1 means always first.
- nDCG: a ranking measure that rewards relevant results near the top, useful when several passages are relevant with different degrees.
Measure at the k you actually send to the model. If you send 5 chunks, recall at 5 is what matters, and also check recall at 30 or 50 to see whether a reranker could recover what's missing. The embedding model page has code for recall at k and MRR.
Generation metrics
These judge the answer, usually with a mix of code checks, an LLM judge and human review.
- Faithfulness (groundedness): is every claim in the answer supported by the retrieved chunks? This is the main hallucination measure for RAG.
- Answer relevance: does the answer address the question that was asked?
- Correctness: does the answer match the expected answer or contain the key facts?
- Citation accuracy: do the cited chunks exist in what was retrieved, and do they support the sentences they're attached to?
- Abstention: for unanswerable questions, does the system say it doesn't know instead of guessing?
- Completeness: for questions with several parts, are all parts answered?
Faithfulness and relevance can be scored by an LLM judge with a clear rubric. Check the judge against your own judgement on a sample before trusting it, as described on the prompt testing page.
Tools
Frameworks like RAGAS, DeepEval and TruLens implement these metrics and the judge prompts for them, and LLM observability platforms can run them on traced production requests. They save time, and their scores are only as good as their judge model and your test set, so understand what each metric measures before you rely on it.
End-to-end and operational metrics
Also track:
- Latency at each stage: retrieval, reranking, generation.
- Cost per question.
- In production: user feedback (thumbs up or down), follow-up questions that suggest the first answer failed, and escalations to humans.
Use the two halves to debug
| Retrieval | Answer | What it means | Where to fix |
|---|---|---|---|
| Correct passage retrieved | Answer correct | Working | |
| Correct passage retrieved | Answer wrong | Generation problem | Prompt, chunk order, number of chunks, model |
| Correct passage not retrieved | Answer wrong | Retrieval problem | Parsing, chunking, embeddings, hybrid search, reranking |
| Correct passage not retrieved | Answer correct | Model answered from memory | Check grounding instructions, this can hide future failures |
| No passage exists | Model guessed | Abstention problem | Prompt and abstention rules |
Run it on every change
Re-run the evaluation whenever you change chunking, the embedding model, the vector index settings, hybrid search weights, the reranker, the prompt, or the LLM. Small changes in early stages can have large effects on answers, and the only way to see them is to measure.
From the floor
On Build30 day 9, while building the RAG pipeline, a builder who worked in testing asked which framework and which metrics would let him show the business that the model's answers were accurate, for example that search should still work when a user misspells a word. My answer was that evaluation is a subject of its own and needs its own session, because the right metrics depend on the use case, and in that build, the pipeline returned the answer, the sources and grounding checks, so every answer could be traced back to the chunks it came from.