How to evaluate RAG in two separate halves
A RAG system has two halves: a search step that finds documents, and a model that writes the answer from them. When an answer is wrong, either the right information never reached the model, or it did and the model misused it, so your evals have to tell you which. Test search with search metrics like recall@k, test generation with checks between the question, the retrieved context and the answer, and always debug search first, because a perfect writer can't fix a missing document.
Two halves, two kinds of failure
| What went wrong | Which half | Example |
|---|---|---|
| The right chunk wasn't retrieved | Search | The 2025 refund policy exists, the 2023 one came back |
| The right chunk was retrieved, the answer ignored it | Generation | Context says 14 days, the answer says 30 |
| The answer added facts not in the context | Generation | Mentions a fee that appears nowhere |
| The question can't be answered from the documents | Both | Should have said "I don't know" |
An overall "answer quality" score mixes these together. Separate scores tell you where to spend your time.
Evaluating search
Search metrics answer one question: did the right chunks come back, and how high?
- Recall@k: of all the relevant chunks for a question, how many appear in the top k results?
- Precision@k: of the top k results, how many are relevant?
- MRR (mean reciprocal rank): how high does the first relevant result rank, on average?
The RAG evaluation page has code for recall@k and MRR.
Building a search test set without manual labels
You can build a search test set from your own corpus:
- Take chunks from your documents.
- Pull out key facts from each one.
- Have an LLM write questions those facts answer, phrased the way users ask.
- Each question's relevant chunk is the one it was generated from.
Review a sample by hand, and add real user questions as soon as you have them, because generated questions are often too close to the document's wording.
Evaluating generation: six checks
Jason Liu treats search metrics as the foundation, then defines six checks between the question (Q), the retrieved context (C) and the answer (A).
The three main ones:
- C|Q, context relevance: is the retrieved context relevant to the question?
- A|C, faithfulness: is every claim in the answer supported by the context?
- A|Q, answer relevance: does the answer address the question?
The three advanced ones:
- C|A, context support: does the context cover everything the answer says?
- Q|C, answerability: can the question be answered from this context at all?
- Q|A, self-containment: does the answer make sense on its own?
When the context can't answer the question (Q|C fails), the right behaviour is to say "I don't know". See play 10.
Most of these checks need an LLM judge. Write each one as a narrow pass or fail judge and validate it against your expert.
Domain-specific failures
Error analysis will also find failures specific to your domain that none of the six checks name. A medical bot mixing adult and child doses, a finance bot using the wrong quarter, a legal bot citing a repealed section. Those need their own checks, and they're often the ones that matter most.
Debug in order
- Check search: was the right chunk in the top k?
- If not, fix retrieval: chunking, embeddings, hybrid search, query rewriting, metadata filters.
- If yes, check generation: faithfulness, relevance, abstention.
- Then fix the prompt or the model.
A real incident
In May 2024, Google's AI Overviews told users to add glue to pizza sauce and to eat one small rock a day. The glue came from a joke Reddit comment, and the rock advice came from a satirical article in The Onion. Search pulled a joke, and the model faithfully repeated it. A faithfulness check would pass that answer, because the answer matched its context. Only a context relevance and source quality check fails it.
From the floor
In the Build30 RAG session, the pipeline had its own evaluate step with a grounding check: remove stop words like "the" and "with" that add no meaning, then check whether the answer's words come from the retrieved chunks, and return the answer with its sources and a grounded flag. A word-overlap check like that is a quick first signal for faithfulness, and it can't catch an answer that reuses the context's words with the meaning reversed, which is where a validated judge comes in.