Long context vs retrieval (RAG): when to use which
If the information fits comfortably in the context window and the task needs the whole thing at once, like summarising one report or comparing two contracts, long context is simpler and often better. If the knowledge base is larger than the window, changes often, is queried many times, or needs access control per user, use retrieval. Most production systems combine them: retrieve the relevant documents, then give the model enough context around them to reason well.
Two ways to give a model knowledge
A model only knows what it was trained on and what you put in front of it. For anything else, like your company's documents, you have two options.
Long context means putting the documents directly into the prompt. Context windows of hundreds of thousands or millions of tokens make this possible for surprisingly large amounts of text.
Retrieval means storing the documents outside, finding the parts relevant to each question, and putting only those into the prompt. That is retrieval-augmented generation.
Where long context wins
- The task needs the whole document. Summaries, overall analysis, consistency checks across a document, and comparisons between two documents need everything, and retrieval would risk missing parts.
- The material is small. A 40 page policy fits easily, and building a retrieval pipeline for it is extra work with no benefit.
- Questions are unpredictable. When users ask about relationships spread across a document, retrieval may not find all the pieces.
- Speed of building. No chunking, embedding, vector database or retrieval tuning.
Where retrieval wins
- The knowledge base is larger than any window. Thousands of documents won't fit, whatever the model.
- Cost and latency at volume. Sending 200000 tokens on every question costs and waits for 200000 tokens every time, while retrieval might send 3000.
- Accuracy on specific questions. Less irrelevant text means fewer distractions, and models are more accurate with focused context than with everything.
- Freshness. When documents change daily, you update the index, not every prompt.
- Access control. Different users can see different documents, and retrieval can filter by permission before anything reaches the model.
- Traceability. Retrieved chunks give natural citations you can show and verify.
Comparison
| Factor | Long context | Retrieval |
|---|---|---|
| Corpus size | Must fit the window | Unlimited |
| Setup effort | Low | Medium to high |
| Cost per query | High, grows with document size | Low, grows with chunks retrieved |
| Latency | Higher (more tokens to process) | Lower for the model call, plus retrieval time |
| Questions needing the whole document | Strong | Weak unless retrieval is designed for it |
| Specific factual questions | Good, can suffer from distraction | Strong if retrieval finds the right chunk |
| Frequently changing data | Rebuild the prompt every time | Update the index |
| Per-user permissions | Hard | Natural with metadata filters |
| Main failure mode | Model misses or confuses details in long input | Retrieval misses the right chunk |
Caching changes the math
Prompt caching reduces the cost and latency of long context when the same documents are used across many requests, for example many users asking about the same handbook. If the long document is identical and at the start of every prompt, cached reads cost much less than normal input. That makes long context practical for some cases that used to need retrieval, as long as the document set is small and stable.
Hybrid approaches
Most real systems use both.
- Retrieve, then expand: retrieve the most relevant chunks, then include the surrounding section or the whole short document they came from, so the model has context around each match.
- Retrieve documents, not chunks: use retrieval to pick the 3 most relevant documents, and put those whole documents in a long context.
- Route by question type: send summary and comparison questions to a long-context path, and specific questions to a retrieval path.
- Agentic retrieval: let an agent search, read whole documents when needed, and search again, see agentic RAG.
How to decide for your case
- Measure your corpus in tokens, with the tokenizer of the model you will use.
- List the types of questions users will ask, and mark which need the whole document.
- Estimate the cost per query for both approaches at your expected volume.
- Build both quickly on a sample and run the same evaluation set through each.
- Compare accuracy, latency and cost, and decide per question type if needed.
From the floor
When we built a question answering system over companies' annual financial reports, retrieval was the natural choice, because a single annual report is long, users ask specific questions like the main risks in the company's international supply chain, and the answer needs to come with the source pages. We loaded the PDF, split it into 1000-character chunks with 200 characters of overlap, stored them in Chroma, retrieved the top 5 chunks for each question, and returned the answer with its source documents.