RAG vs Long-Context Models: When to Retrieve and When to Stuff the Prompt

Long-context models let you paste tens or hundreds of thousands of words straight into a prompt, so for a one-off question on a handful of documents, that's often the simplest option. But once you query the same material repeatedly, your document set grows past a few hundred pages, or you need citations to a specific paragraph, retrieval (RAG) tends to be cheaper, faster and more accurate. The right choice depends on how big your corpus is and how often you ask questions of it, not on which approach sounds more modern.

What a context window actually is

A context window is the maximum amount of text (measured in tokens, roughly three-quarters of a word each) that a model can read in a single request, covering both your prompt and its reply. Modern models offer windows ranging from around 32,000 tokens up to well over a million, which sounds like plenty of room for most business documents. The catch is that every token in that window gets processed, and billed, on every single call, whether or not it was relevant to the question asked.

Tokens, and why they cost money twice over

Providers charge per token for input (what you send) and output (what the model generates), and input tokens from a huge pasted document are rarely free just because the model didn't need most of them to answer. If you paste a 300-page operations manual into every question, you pay for processing that manual every time, even if the answer lives on one page. That cost scales with the size of your corpus and the number of questions you ask, which is exactly the combination that makes long-context stuffing expensive at scale.

What RAG does differently

Retrieval-augmented generation splits the problem in two: an indexing step that chunks and embeds your documents once, and a query step that fetches only the handful of passages most relevant to the question before asking the model to answer. The model never sees the whole corpus, only the few paragraphs that matter for this particular query, which keeps the prompt small regardless of how large the underlying document set grows.

1. Documents are split into chunks (a paragraph, a clause, a section) sized to preserve meaning without pulling in unrelated text.
2. Each chunk is indexed for both keyword and semantic (meaning-based) search, so queries find relevant text even when the wording differs from the source.
3. A question triggers a search across the index, returning the top few matching passages.
4. The model generates an answer grounded in those passages, often with citations pointing back to the exact source.

If you want the full mechanics of that pipeline, we walk through it step by step in How RAG Works, Step by Step, following one question from index to answer.

When stuffing the whole document in the prompt works

Long context genuinely wins in several situations, and it's worth being honest about them rather than defaulting to retrieval out of habit.

• A single session, one-off analysis: summarising one contract, reviewing one report, or asking follow-up questions about one PDF you've just opened.
• Small, stable corpora: a handful of documents that rarely change and that genuinely fit within one context window with room to spare.
• Tasks that need the whole document held together: cross-referencing clause 4 against clause 19 of the same contract, or spotting inconsistencies that span an entire text, where chunking could lose the connective thread.
• No need for citations or an audit trail, and no requirement to query the same material from multiple tools or team members over time.

When retrieval is cheaper, faster and more accurate

Retrieval becomes the stronger option as soon as any of these apply: the corpus is large (hundreds of documents, policy manuals, years of case notes), the same material gets queried repeatedly by different people, you need the answer to cite the exact source passage, or the documents keep changing and you don't want to re-paste them every time. In each case, long-context stuffing either costs too much per question, runs too slowly, or produces answers that are hard to verify.

Cost per query: a worked comparison

The clearest way to see the trade-off is to compare how cost behaves as the corpus and the number of queries grow. Long-context cost scales with corpus size multiplied by query volume, because the full text is resent every time. RAG cost scales mainly with query volume, because indexing happens once and retrieval only fetches a few chunks per question.

In practice, the break-even point arrives sooner than most teams expect. Once a knowledge base is queried more than a few dozen times, or once the underlying documents exceed roughly a few hundred pages, retrieval almost always comes out cheaper and faster, even before counting the benefit of citations.

Accuracy and verifiability, not just speed

Cost isn't the only reason teams move to retrieval. Grounded answers with cited passages let a reader check the source in seconds, which matters for anything regulatory, contractual or clinical. Good retrieval also depends on how the text was indexed in the first place: combining keyword matching with semantic search catches both exact terminology and paraphrased questions, which is why we cover the mechanics in Hybrid Search: Why BM25 Plus Vectors Beats Either Alone for RAG. How documents are split into chunks matters just as much, since chunks that are too small lose context and chunks that are too large dilute relevance. We go into that trade-off in Chunking Strategies for RAG.

A practical decision checklist

Before building anything, run through these questions. They'll usually settle the debate faster than benchmarking two approaches side by side.

• How big is the document set now, and how big will it be in a year? Static and small favours long context; large and growing favours RAG.
• How many questions will be asked per day or per month? A handful favours long context; dozens or hundreds favour RAG on cost alone.
• Do answers need to cite a specific clause, page or section? If yes, retrieval with grounded citations is the safer default.
• Will documents be updated regularly? If so, re-indexing changed files is cheaper than re-pasting an entire corpus every session.
• Will more than one tool or person need to query the same material? A shared, queryable knowledge base avoids duplicating the same large prompt everywhere.

Many teams end up doing both: long context for ad hoc, single-document deep dives, and a retrieval-backed knowledge base for anything queried repeatedly. If you're building the latter, Kopik lets you upload documents, have them chunked and indexed for hybrid search automatically, and query the result from the website, a REST API, or an MCP server usable from Claude, Cursor and ChatGPT, without having to build the retrieval pipeline yourself.

Building it yourself vs using RAG as a service

If you'd rather wire retrieval into your own application, Adding a Knowledge Base API to Your App: A Developer's Guide covers the integration side, and the full developer documentation lists the API and MCP endpoints. If you'd rather not run the infrastructure at all, browsing the catalogue of ready-made knowledge bases is worth a look before building from scratch, since someone may have already indexed the regulatory or technical material you need.