Under the hood

Agentic RAG: When AI Agents Search for Themselves

The Kopik team9 min read

Agentic RAG is retrieval where a language model, not a fixed pipeline, decides what to search for, when to search again, and which knowledge base to call next. Instead of one query producing one set of passages, the agent plans a query, retrieves, evaluates what came back, and decides whether to refine, call a different source, or answer. It earns its keep when a single question genuinely needs several lookups across different documents or bases. It is also slower, harder to debug and more expensive per answer than standard RAG, so it is worth understanding the trade-offs before you build one.

Agentic RAG vs. standard RAG: what's actually different

A standard retrieval-augmented generation pipeline, as described in What Is RAG, is one shot: embed the question, retrieve the top-k chunks, hand them to the model, get an answer. It is fast, predictable and cheap because every question costs exactly one retrieval call and one generation call. Agentic RAG wraps that retrieval step as a tool the model can call repeatedly, with reasoning in between calls. The model sees the first batch of passages, decides whether they actually answer the question, and if not, reformulates the query or splits it into sub-questions before retrieving again. How RAG Works, Step by Step walks through the single-shot version; agentic RAG is that same loop run two, three or more times with a planning step wrapped around it.

The distinction matters because the two approaches solve different problems. Standard RAG answers questions that live inside one or two well-matched chunks. Agentic RAG answers questions that require combining facts scattered across several documents, comparing two sources, or chasing down a term the first retrieval attempt missed entirely.

How multi-step retrieval and query planning work

In practice, an agentic RAG loop looks like a small state machine the model drives itself. The model receives the user's question, and instead of immediately generating an answer from whatever the first search returns, it treats retrieval as one tool among several, to be called as many times as needed.

  1. Decompose the question into sub-questions if it bundles more than one request
  2. Retrieve for the first sub-question using an initial query
  3. Assess whether the returned passages actually answer it, not just whether they are topically related
  4. Reformulate the query if retrieval was weak, trying different keywords, a broader phrasing, or a narrower one
  5. Decide whether a remaining sub-question needs a separate lookup, possibly in a different knowledge base
  6. Stop once confidence is high enough or a step budget is reached
  7. Compose the final answer, citing the specific passages pulled from each retrieval call

That last step is where a lot of agentic RAG implementations quietly fall apart: if the agent doesn't keep track of which passage came from which step, the final citation trail gets muddled. Grounding each sub-answer in its own cited passages, the way trustworthy AI answers should be built, keeps the chain auditable even when there are four or five steps behind it.

Calling multiple knowledge bases as tools

The more interesting (and more fragile) version of agentic RAG involves not just repeated calls to one source, but an agent choosing between several knowledge bases as distinct tools. A support agent might have access to a product documentation base, a billing policy base and a past-tickets base, and decide per sub-question which one to query. This is the pattern behind why AI agents need reliable knowledge sources: the agent is only as good as its ability to pick the right source and recognize when none of them actually answer the question.

The Model Context Protocol is a common way to wire this up in practice: each knowledge base is exposed as an MCP tool with its own description, and a client like Claude, Cursor or a custom agent framework picks which tool to call based on the sub-question's content. Model Context Protocol (MCP) Explained covers the mechanics, and Connect a Knowledge Base to AI Agents with MCP walks through wiring a base up as a callable tool.

Agent orchestration patterns

PatternHow it routesExample use case
Single base, multi-stepSame base queried repeatedly with refined queriesA research question that needs several passages to fully answer
Router agentPicks one base per sub-question based on topicSupport bot routing between product docs and a billing policy base
Parallel fan-outQueries several bases at once and merges resultsComparing pricing and terms across multiple public knowledge bases
Sequential chainOutput of one query informs the next query to a different baseLegal research: look up a statute, then search case notes that cite it

The risks nobody mentions in the demo

Agentic RAG demos well because the examples are picked to show the agent recovering from a bad first search. In production, the same looping behavior creates problems that a single-shot pipeline simply doesn't have.

  • Cost multiplication: each step is a retrieval call plus a model call, so a five-step agent can easily cost five to ten times a single RAG query
  • Latency: multi-step loops take seconds to tens of seconds end to end, which is a poor fit for synchronous chat interfaces where users expect a near-instant reply
  • Compounding errors: if step two misreads the passages from step one, that mistake carries into later queries and shows up in the final answer with false confidence
  • Loops that don't converge: a poorly bounded agent keeps "refining" its query without ever reaching a stop condition, burning calls without producing a better answer
  • Harder auditing: a single citation trail becomes several, and tracing which step produced which claim takes real discipline, see How to Evaluate a RAG System
  • Over-retrieval: an agent with access to many bases sometimes calls ones it doesn't need, adding cost and noisy, irrelevant context to the final prompt

Set a hard step budget

Cap the agent at three or four retrieval steps per question, log every call it makes (query, source, result count), and have it return its best answer with a caveat rather than keep searching indefinitely. A bounded, slightly imperfect answer beats an unbounded loop that never terminates.

What agentic RAG actually costs

A standard RAG answer is roughly one embedding call, one retrieval call and one generation call. Agentic RAG multiplies all three: every extra step adds a retrieval cost (credits, if you are querying paid bases) and a generation cost, because each step's passages get fed back into the model's context for the next reasoning pass. What Does RAG Really Cost? breaks down the single-shot numbers; the table below shows roughly how those numbers move once an agent starts looping.

Cost comparison: standard vs. agentic RAG

ComponentStandard RAG (1 call)Agentic RAG (avg. 4 steps)
Retrieval calls13 to 5
Model generation calls14 to 6 (reasoning plus final answer)
Tokens processedBaseline3x to 5x baseline
Typical latency1 to 3 seconds8 to 30 seconds
Best fitFAQ, single-document lookupMulti-document research, cross-base comparison

When you need agentic RAG, and when you don't

Most questions that come into a support widget, an internal wiki assistant or a documentation search box don't need multi-step retrieval at all. Agentic RAG earns its cost in a narrower set of cases.

  • Use it when a question genuinely spans multiple documents or bases that no single retrieval pass can cover, such as comparing this quarter's filing to last year's and to an internal risk policy
  • Use it when the first retrieval attempt is likely to miss because of ambiguous terminology, acronyms, or sources in different languages, see Multilingual RAG
  • Skip it for simple factual lookups, FAQ-style support or single-document Q&A, where standard RAG is cheaper, faster and far easier to audit, as in RAG for Customer Support or Adding AI Search to Your Website
  • Consider a long-context model instead of a multi-step agent when the whole corpus fits in the context window and the source material changes rarely, see RAG vs Long-Context Models

Building agentic RAG with Kopik

The practical building block for agentic RAG is a knowledge base that answers reliably on its own, with citations, so that whatever is orchestrating the agent can trust each individual tool call. Each base created on Kopik, whether built from PDFs, Word files or Markdown through the base creation page, can be queried through a REST API or an MCP server, which means an agent running in Claude, Cursor or a custom framework can call a specific base, read grounded passages back, and decide whether to call another base before composing its final answer. Because each response stays grounded in cited source text rather than free-form generation, the kind of traceability discussed in Building AI Answers People Can Trust holds even across a multi-step chain.

For teams running several domains of knowledge, product docs, HR policy, legal memos, it's usually cleaner to keep them as separate bases in the catalogue and let the agent's orchestration layer route between them, rather than merging everything into one base and hoping retrieval sorts it out. That separation also keeps pricing predictable: credits are consumed per question per base, so a step budget on the agent side doubles as a cost control.

Give your agents retrieval tools

Turn your documents into queryable knowledge bases and expose them to AI agents through a REST API or an MCP server.

A short checklist before you ship one

  1. Cap the number of retrieval steps per question, and log each step's query and source
  2. Track cost per answer, not just per retrieval call, since agentic loops multiply both
  3. Keep citations attached at every step so the final answer can be traced back to specific passages
  4. Prefer separate, well-scoped knowledge bases over one giant base an agent has to guess inside
  5. Fall back to a single-shot answer with a caveat when the step budget runs out, instead of failing silently

Frequently asked questions

What is agentic RAG in simple terms?

It's retrieval-augmented generation where the model itself decides what to search for and when to search again, instead of a fixed pipeline running one retrieval pass per question. The model plans queries, evaluates results, and loops until it has enough to answer.

How is agentic RAG different from an agent that just calls one search tool?

An agent that calls a search tool once and answers is functionally the same as standard RAG with extra steps. Agentic RAG specifically involves the model deciding whether to search again, reformulate the query, or call a different source based on what the first search returned.

Does agentic RAG reduce hallucinations?

It can, when the extra steps genuinely fill gaps a single search missed. But it can also compound errors: if an early step misreads its passages, that mistake carries into later queries and the final answer. Citations at every step are what keep this in check.

How many retrieval steps is too many?

There's no universal number, but most practical implementations cap agents at three to five steps per question. Beyond that, cost and latency grow quickly while the odds of the loop actually converging on a better answer drop.

Can agentic RAG query multiple knowledge bases at once?

Yes. A common pattern exposes each knowledge base as a separate tool, often through MCP, and lets the agent route each sub-question to the base most likely to answer it, or fan out to several bases in parallel and merge the results.

Is agentic RAG worth the extra cost for a small team?

Usually not for simple FAQ or single-document lookups, where standard RAG is cheaper, faster and easier to audit. It becomes worth it when questions routinely require combining facts from multiple documents or sources that a single retrieval pass can't cover.

Get the Kopik newsletter

New knowledge bases, RAG guides and product news. One email every week or two, unsubscribe in one click.

By subscribing you agree to receive our newsletter. We never share your address.