Agentic RAG: when retrieval needs to plan
A standard RAG pipeline always does the same thing: retrieve once, then answer. Agentic RAG turns retrieval into tools that an agent can use as many times as it needs: it decides whether to search, which source to search, reads the results, notices what's missing, and searches again before answering. It handles complex, multi-step and multi-source questions much better, and it is slower, more expensive and less predictable, so use it for the questions that need it and keep a fixed pipeline for the rest.
RAG, agents and agentic RAG
These three get mixed up, so it helps to separate them.
An LLM answers from what it learned in training. RAG is a strategy that connects an LLM to external data: retrieve relevant information, then generate the answer from it. An agent uses an LLM to decide on actions and executes them with tools.
Agentic RAG puts them together: retrieval becomes one or more tools, and an agent decides how to use them.
What a fixed pipeline can't do
A fixed pipeline retrieves once, with the question as written. It can't:
- Decide that a greeting or a general question needs no retrieval at all.
- Choose between sources, like the HR documents, the product database or the web.
- Notice that the first search returned nothing useful and try different words.
- Break a complex question into steps where each answer shapes the next search.
- Combine documents with live data from an API.
How agentic RAG works
The agent gets tools like these:
search_policies(query): hybrid search over the policy documents.search_tickets(query, customer_id): search past support tickets.get_order(order_id): look up an order in the database.web_search(query): search the internet.
Then it works in a loop, the reason, act, observe loop of the ReAct pattern:
- Read the question and decide what information is needed.
- Call a search tool with a query it writes.
- Read the results and decide whether they are enough.
- If not, refine the query, try another source, or look up a specific record.
- When it has enough, answer with citations, or say what it couldn't find.
Patterns
- Routing: one model call decides which source or pipeline to use, then a fixed pipeline runs. The simplest step toward agentic RAG, and often enough.
- Iterative retrieval: search, check, refine and search again, with a limit on the number of rounds.
- Self-checking retrieval: the model grades whether retrieved chunks are relevant, and retrieves again or falls back to another source if not. Research approaches like Self-RAG and Corrective RAG (CRAG) work this way.
- Multi-step research: plan sub-questions, answer each with retrieval, then combine. Deep research tools are a large version of this.
Costs and risks
- Latency: several model calls and searches, so answers take seconds to tens of seconds.
- Cost: every step resends the growing context, so a single question can cost many times a fixed pipeline.
- Unpredictability: the agent may take different paths for similar questions, which makes testing harder.
- Loops: without limits, an agent can keep searching without converging.
- Context growth: search results pile up, and long context hurts accuracy.
So put limits in place: a maximum number of steps, a token or cost budget per question, a time limit, and results trimmed before they go back into context. See agent cost control.
When to use it
Start with a fixed pipeline and add agency where your evaluation shows it is needed.
| Question type | Approach |
|---|---|
| Simple factual questions | Fixed RAG pipeline |
| Questions that may not need retrieval, or need a different source | Routing |
| Questions the first search often misses | Iterative retrieval with a step limit |
| Complex, multi-part, multi-source questions | Agentic RAG |
| Long research reports | Multi-step research agent, run as a background job |
A common production design routes simple questions to the fast pipeline and complex ones to the agent.
From the floor
In the RAG agent workshop, I put the three ideas side by side before building: LLMs answer from their trained parameters, RAG connects them to external data, and agents can execute tasks with tools, and you can use LLMs and RAG inside agents. The build was a document question answering agent over PDFs, the starting point for an agent that searches documents as one of its tools.