Agentic RAG: When AI Agents Search for Themselves
Agentic RAG is the pattern where an AI agent does not just retrieve once and answer: it plans its own searches, decides which knowledge base to query, reads what comes back, and decides whether to search again before replying. It turns retrieval from a single lookup into a reasoning loop that can span several tools and several rounds of queries. This is powerful for complex, multi-part questions, but it also multiplies cost, latency and the chances of something going wrong. This article explains how multi-step retrieval and query planning actually work, what happens when an agent can call several knowledge bases, and how to keep the whole thing reliable and affordable.
What makes RAG agentic, and what doesn't
Classic RAG, the kind described step by step in How RAG Works, Step by Step, is single-shot: the user asks a question, the system embeds the query, retrieves the top matching chunks, and generates an answer grounded in those chunks in one pass. Agentic RAG adds a decision loop on top of that. The agent, usually a language model running with a system prompt and a set of callable tools, decides for itself whether retrieval is even needed, what exactly to search for, which source to use, and whether the results it got back are good enough to answer from. It can call a search tool several times, rewrite its own query, or switch to a different knowledge base entirely, before it ever writes a final answer. The line between the two is not always sharp: a system that retrieves once, checks if the answer is complete, and retrieves a second time to fill a gap is already behaving agentically, even if it never touches more than one base.
From one-shot retrieval to multi-step search
Query decomposition and planning
Compound questions rarely map cleanly onto a single search. A question like 'compare the termination clause in this supplier contract with the one in last year's version and flag anything riskier' really contains two or three separate lookups plus a comparison step. An agentic system handles this by first producing a plan: a short list of sub-queries, in an order that makes sense, often written out by the model itself before any retrieval happens. Each sub-query is then run against the knowledge base, or bases, as a normal search, and the results are fed back into the model's context for the next step. This planning step is what separates agentic RAG from simply retrieving a larger number of chunks: the agent is reasoning about what it still needs to know, not just pulling more text.
Iterative retrieval: search, read, search again
The core loop looks like this: search, read the returned passages, decide if the question can now be answered, and if not, reformulate the query and search again. Reformulation might mean narrowing the keywords, widening a date range, trying a synonym the first search missed, or asking a more specific follow-up question once an initial, broader search has given the agent enough context to know what to ask next. A well-built agent also runs a lightweight self-check before answering: does every claim in the draft answer trace back to a retrieved passage, or is it filling gaps with assumption? This is the same faithfulness question covered in How to Evaluate a RAG System: Retrieval Metrics, Faithfulness and Citation Accuracy, it just gets checked mid-conversation instead of only at evaluation time.
Calling several knowledge bases: tools, not one big index
The more interesting case, and the one that genuinely needs an agent rather than a single retrieval call, is when a question spans sources that live in separate knowledge bases. Rather than merging everything into one giant index, each base is exposed to the agent as a distinct tool with its own name and description, for example a search over an HR handbook, a separate search over supplier contracts, and a third over regulatory guidance. The agent reads the question, decides which tool or tools are relevant, calls them (sometimes in sequence, sometimes in parallel), and merges the results before writing an answer. This is exactly the shape of integration that the Model Context Protocol was designed for: an MCP client such as Claude, Cursor or ChatGPT can see several knowledge bases as separate tools and pick between them at runtime, without you hardcoding routing logic. Connect Claude, Cursor or ChatGPT to a Knowledge Base walks through the setup if you want an agent querying a Kopik base directly from your editor or chat client.
- A customer support agent checking both a product manual base and a separate returns-and-refunds policy base before replying to a ticket
- A legal assistant checking a statute base alongside a firm's internal memo base, as described in RAG for Law Firms and Legal Teams
- An accountant's agent checking HMRC guidance in one base and a client's own working papers in another before answering a query about an allowance
- A multilingual support desk routing a French question to a French-language base and an English one to an English base, rather than one mixed index
What can go wrong when agents search for themselves
Letting a model decide when and what to search is powerful, but it removes a layer of predictability that simple RAG gives you almost for free. The failure modes are different in kind, not just degree.
Agentic RAG risks and mitigations
| Risk | Why it happens | Mitigation |
|---|---|---|
| Redundant or infinite search loops | The agent keeps reformulating without converging on an answer | Hard cap on retrieval steps per question, forced answer after the limit |
| Wrong tool selection | Tool names or descriptions are ambiguous, agent picks the wrong base or none | Write precise, non-overlapping tool descriptions; test routing explicitly |
| Compounding hallucination | An early misread passage becomes the false premise for later steps | Require a citation for every claim at each step, not only in the final answer |
| Prompt injection via retrieved content | Instructions hidden inside a document steer the agent's next tool call | Treat retrieved text as untrusted data, never as instructions to execute |
| Latency stacking | Each round trip adds seconds; a multi-step chain can take 10 to 30 seconds | Run independent sub-queries in parallel where possible, show progress to the user |
| Unpredictable cost per question | The number of calls varies with how the agent decides to search | Set a hard credit or token budget per question, not only per month |
Log the plan, not just the answer
If you only store the final response, you cannot tell whether the agent searched the right source for the right reason. Keep a record of each sub-query, which base it hit, and which passages came back, so you can audit a wrong answer afterwards rather than guessing.
The cost of agentic RAG
Every extra step in an agentic chain is at least one more retrieval call and one more generation call, and generation calls also grow in size because each step carries the accumulated context of the steps before it. A question answered in a single shot might cost roughly one retrieval plus one generation; the same question handled agentically, with a plan, two reformulated searches and a final synthesis, can realistically cost three to five times as much, before you even add a second knowledge base into the mix. What Does RAG Really Cost? Ingestion, Storage, Retrieval and Generation Explained breaks down the single-shot numbers; agentic chains simply multiply the retrieval and generation line items by the number of steps actually taken, which is often only known after the fact.
Rough cost multiplier by retrieval pattern
| Pattern | Retrieval calls | Generation calls | Relative cost vs single-shot |
|---|---|---|---|
| Single-shot RAG, one base | 1 | 1 | 1x |
| Agentic RAG, one base, self-correcting | 2-3 | 2-3 | 2-3x |
| Agentic RAG, two or more bases, with reformulation | 4-8 | 4-8 | 4-8x |
If you query Kopik bases through prepaid credits or the €12 per month chat subscription, that cost is predictable per base, because hybrid search already combines full-text and semantic matching in a single call, so coverage is usually strong without needing several reformulated searches. The unpredictability mostly shows up when you build the orchestration yourself against a raw LLM API and let the agent decide how many times to call out. It is also worth checking whether you need retrieval at all for a given question: RAG vs Long-Context Models: When to Retrieve and When to Stuff the Prompt covers the cases where a long context window is simply cheaper than any retrieval loop.
Designing agentic RAG that stays reliable
- Cap the number of retrieval steps, typically three to five, and force a final answer once the cap is reached rather than letting the loop run indefinitely
- Require a citation at every intermediate step, not only in the final answer, so each sub-query's contribution can be checked
- Log the full plan and every tool call for later review, using faithfulness and citation accuracy checks like those in How to Evaluate a RAG System
- Treat all retrieved content as untrusted data: strip or ignore anything inside a document that looks like an instruction aimed at the agent itself
- Keep search tools separate from action tools; an agent should never write, send or delete something based purely on retrieved content without a human check
- Start with a single knowledge base before letting the agent choose between several, then add tools one at a time once routing is proven correct
- Set a hard credit or token budget per question so a single unusually complex query cannot silently consume a month's allowance
When you don't need an agent at all
Most real questions are not actually compound. A single, well-posed query against a well-chunked, hybrid-searchable knowledge base answers the majority of support tickets, policy lookups and document questions directly, in one pass, with one set of citations. Agentic loops earn their cost when questions genuinely span sources, require comparison across documents, or need the system to notice a gap and go looking for the missing piece on its own. If you are not sure which camp a use case falls into, Why AI Agents Need Reliable Knowledge Sources is a good place to check what 'reliable' actually requires before you add planning and tool choice on top. For straightforward one-base querying, building or trying a base from the catalogue, or creating your own at /dashboard/bases/new, will usually get you a working, citable answer faster than standing up an agent framework.
Give your agent real knowledge bases to search
Expose one or more Kopik knowledge bases as tools through the REST API or the MCP server, and let your agent plan its own queries across them, with citations on every answer.
Frequently asked questions
What is the difference between RAG and agentic RAG?
Classic RAG retrieves once and generates an answer from what it finds. Agentic RAG adds a decision loop: the agent plans its own sub-queries, decides which knowledge base to search, checks whether the results are sufficient, and can search again before answering.
Does agentic RAG always need multiple knowledge bases?
No. An agent can behave agentically against a single base, for example by reformulating a query when the first search comes back thin. Multiple bases become useful when a question genuinely spans separate sources, such as a product manual and a returns policy.
How many retrieval steps should an agent be allowed?
Most practical implementations cap the loop at three to five steps and force a final answer once the limit is reached. Uncapped loops risk burning credits on redundant searches without ever converging.
Can agentic RAG hallucinate more than classic RAG?
It can, because an early misread passage becomes the premise for later planning steps, compounding the error. Requiring a citation at every intermediate step, not just the final answer, makes this easier to catch.
Is agentic RAG more expensive than classic RAG?
Usually yes. Each extra step is another retrieval call and another generation call carrying accumulated context, so a multi-step chain can realistically cost three to eight times a single-shot answer depending on how many bases and reformulations are involved.
Can I build agentic RAG with MCP?
Yes. The Model Context Protocol lets an MCP client such as Claude, Cursor or ChatGPT see several knowledge bases as separate callable tools and choose between them at runtime, which is exactly the tool-selection step agentic RAG relies on.
Get the Kopik newsletter
New knowledge bases, RAG guides and product news. One email every week or two, unsubscribe in one click.
By subscribing you agree to receive our newsletter. We never share your address.