How to Evaluate a RAG System: Retrieval Metrics, Faithfulness and Citation Accuracy
Evaluating a RAG system means measuring two separate things: whether retrieval surfaces the right passages (recall@k, MRR, precision) and whether the generated answer is faithful to those passages (faithfulness, citation accuracy). You need both a one-time test set of real questions and ongoing production monitoring, because a system that scores well on day one can silently degrade as documents change.
Why RAG evaluation is not the same as LLM evaluation
A retrieval-augmented generation system has two failure points, not one. The retrieval step can miss the right document entirely, or rank it too low for the model to use. The generation step can retrieve the right passage and still misquote it, blend it with unrelated training knowledge, or cite the wrong source. Evaluating only the final answer with a generic quality score hides which stage is broken, and that matters because the fixes are completely different: better chunking and hybrid search for retrieval problems, versus stricter prompting or a 'passages only' mode for generation problems.
If you are new to the mechanics of retrieval-augmented generation, it helps to start with What Is RAG (Retrieval-Augmented Generation)? A Practical Guide and How RAG Works, Step by Step before setting up an evaluation pipeline, since the metrics below map directly onto the pipeline stages.
Retrieval metrics: recall@k, MRR, and precision
Retrieval metrics answer a narrow question: for a given query, did the search step return the chunk that actually contains the answer, and how high up the ranking was it? These metrics require a labeled test set where each question is paired with the identifier of the chunk (or chunks) that should be retrieved.
Core retrieval metrics for RAG
| Metric | What it measures | How to read it |
|---|---|---|
| Recall@k | Whether at least one relevant chunk appears in the top k results | Recall@5 of 0.92 means 92% of queries found a relevant chunk in the first 5 results |
| Precision@k | What fraction of the top k results are actually relevant | Low precision with high recall usually means your chunks are too large or too generic |
| MRR (Mean Reciprocal Rank) | How high the first relevant result ranks, averaged across queries | MRR close to 1.0 means relevant chunks usually rank first; 0.3 means users often scroll past noise |
| nDCG | Ranking quality when some chunks are more relevant than others | Useful when you have graded relevance labels, not just yes/no |
In practice, recall@k is the metric to watch first. If the right passage never makes it into the top 5 or top 10 results, no amount of prompt engineering will fix the answer downstream. Low recall almost always traces back to one of three causes: chunks that split ideas across boundaries, embeddings that miss domain-specific vocabulary, or a search method that relies on pure vector similarity and misses exact keyword matches (acronyms, part numbers, statute references). That third cause is why combining full-text search with semantic expansion, as described in Hybrid Search: Why BM25 + Vectors Beats Either Alone for RAG, tends to lift recall more reliably than swapping embedding models.
If retrieval metrics are weak, revisit how documents are split before touching the generation prompt. Chunking Strategies for RAG: Fixed-Size, Semantic, and Structure-Aware Splitting Compared walks through the tradeoffs between fixed-size windows, semantic splitting, and structure-aware chunking, and how each one affects recall@k on real document sets.
Answer faithfulness: does the output match the sources?
What faithfulness actually measures
Faithfulness (sometimes called groundedness) measures whether every factual claim in the generated answer is supported by the retrieved passages, as opposed to the model's own training data. A faithful answer can still be incomplete or awkwardly phrased, but it should never state something the sources do not say. This is the metric that catches hallucinations specifically introduced at the generation step, independent of whether retrieval did its job.
The standard way to score faithfulness without manual review for every answer is to break the generated response into individual claims, then check each claim against the retrieved passages using a separate verification pass (either a human reviewer on a sample, or an automated judge prompt that only sees the claim and the source text, not the original question). The score is the fraction of claims that are directly supported. Treat any claim that requires inference beyond the text as unsupported, since that is exactly the kind of subtle drift that erodes user trust over time.
Separate faithfulness from correctness
A response can be faithful to the wrong passage (if retrieval picked an outdated document) and still be factually wrong in the real world. Score faithfulness against what was retrieved, and score retrieval separately against what should have been retrieved. Conflating the two makes debugging much harder.
For more on why hallucinations happen and how grounding reduces them, see How to Reduce AI Hallucinations With Grounded Answers.
Citation accuracy: can users verify the answer?
Citation accuracy is a narrower, easier-to-automate check than full faithfulness: for each cited passage attached to an answer, does that passage actually contain the information attributed to it? This matters for user trust independently of faithfulness, because a user who can click through to a cited source and verify the claim in ten seconds will trust the system far more than one who has to take the answer on faith. Two failure modes are common here: citing a passage that is topically related but does not actually support the specific claim, and citing the right document but the wrong section or page.
- Spot-check a random sample of answered questions weekly and confirm each citation supports its adjacent claim
- Flag answers with zero citations in 'grounded answer' mode as a hard failure, not a warning
- Track the average number of citations per answer; a sudden drop often signals a retrieval or indexing regression
- Test citation accuracy separately in 'passages only' mode versus 'grounded answer' mode, since they fail differently
If you want readers or team members to understand how to sanity-check cited answers themselves, Document Q&A With Citations: How to Check an AI Answer is a good reference to point them to.
Building a test set of real questions
Synthetic test questions generated by an LLM tend to be too clean: they use the same vocabulary as the source document, which makes retrieval look better than it will be in production. A useful test set is built from questions real users actually ask, which are messier, use different phrasing, sometimes bundle two questions into one, and occasionally ask about something the documents do not cover at all.
- Pull 30-50 real questions from support tickets, chat logs, or a pilot group, if you have them
- For each question, manually identify which chunk(s) should answer it, and write the ideal short answer
- Include 10-15% 'no answer' questions where the correct behavior is to say the documents do not cover this, to catch over-confident hallucination
- Add a handful of adversarial questions with similar wording but different intent (e.g. two product names that differ by one digit) to stress-test retrieval precision
- Re-run the full test set after every change to chunking, the embedding model, or the prompt, and track metrics over time rather than as a one-off score
This test set becomes your regression suite. Store it alongside the documents it refers to, and re-score it whenever the underlying knowledge base is re-indexed, since chunk boundaries and identifiers can shift even when the source text does not.
Continuous monitoring once the system is live
A test set catches regressions you introduce yourself. It will not catch problems that emerge from how the document set evolves in production: new PDFs added without review, stale policies that contradict newer ones, or a spike in questions about a topic the knowledge base barely covers. Continuous monitoring closes that gap.
What to monitor in production
| Signal | Why it matters | Suggested check frequency |
|---|---|---|
| Zero-result or low-confidence queries | Reveals gaps in document coverage before users complain | Weekly review of the top 20 |
| Citation count per answer | Sudden drops indicate an indexing or retrieval regression | Daily automated alert on threshold breach |
| User follow-up or rephrasing rate | High rephrasing suggests the first answer missed the intent | Weekly trend |
| Sampled faithfulness score | Catches generation drift after a prompt or model change | Weekly sample of 20-30 answers |
| Document freshness | Flags documents that haven't been updated despite a known policy change | Monthly audit against source-of-truth owners |
Monitoring does not need heavy infrastructure. A simple dashboard that logs every question, the retrieved chunk IDs, the citations shown, and a thumbs up/down from users is enough to run the weekly checks above. If you are building or exposing this kind of knowledge base through an API, Adding a Knowledge Base API to Your App: A Developer's Guide covers how to log and expose that data alongside the query endpoint, and the developers documentation details the request and response fields you can capture for evaluation, including cited passages and confidence signals returned with each answer.
Putting it together without building a custom pipeline
Teams often delay evaluation because building the retrieval pipeline, the chunking logic, and the citation tracking from scratch is already a full project before you even get to measuring it. One way to shortcut that is to start from a platform where chunking, hybrid search, and cited answers are already handled, so your evaluation work can focus on the test set and the metrics rather than re-implementing retrieval. On Kopik, every knowledge base already returns grounded answers with cited passages or raw passages in 'passages' mode, which means citation accuracy and faithfulness checks can be run directly against the API response without extra instrumentation.
If you want to try this on your own documents, create a knowledge base for free, upload a representative sample, and run your test questions against it through the website or the REST API described in the developers docs. Comparing recall@k and citation accuracy before and after a chunking change takes minutes once the base is set up, which makes it practical to iterate on evaluation rather than treating it as a one-time audit.
Test your documents against real questions
Upload a sample of your documents, run your test set, and check recall and citation accuracy before you commit to a chunking strategy.
A minimal evaluation checklist
- A labeled test set of 30-50 real questions with expected chunks and answers, including 'no answer' cases
- Recall@k and MRR tracked on every chunking or embedding change
- A faithfulness score sampled weekly on live traffic, not just the test set
- Citation accuracy spot-checked against the actual passages shown to users
- A production dashboard logging zero-result queries and citation counts
- A monthly document freshness audit against whoever owns the source content
None of these require exotic tooling. What they require is treating evaluation as a recurring habit tied to every change to the document set or the pipeline, rather than a report generated once before launch and never revisited.
Related reading
For a deeper look at the tradeoff between retrieval and simply pasting documents into a long context window, see RAG vs Long-Context Models: When to Stuff the Prompt and When to Retrieve. And if your evaluation reveals that users query from Claude, Cursor, or ChatGPT rather than a web UI, How to Connect Claude, Cursor or ChatGPT to a Knowledge Base explains how citations and passages surface through an MCP client, which changes how you should sample and review them.
Frequently asked questions
What is a good recall@k for a RAG system?
There is no universal number, but many production systems aim for recall@5 or recall@10 above 0.85-0.90 on their test set. What matters more than the absolute value is tracking it over time: a drop after a chunking or embedding change signals a regression worth investigating before it reaches users.
What is the difference between faithfulness and accuracy in RAG evaluation?
Faithfulness measures whether the answer matches what the retrieved passages say, regardless of whether those passages are correct or current. Accuracy measures whether the answer matches real-world fact. A system can be faithful to an outdated document and still be factually wrong, which is why both need separate checks.
How many test questions do I need to evaluate a RAG system?
A starting set of 30-50 real questions, each labeled with the expected source chunk, is usually enough to catch major regressions in retrieval and faithfulness. Larger knowledge bases with multiple document types benefit from 100 or more, including a share of adversarial and 'no answer' questions.
Can I automate faithfulness scoring without a human reviewer?
Yes, by breaking the answer into individual claims and checking each one against the retrieved passages with a separate verification prompt that only sees the claim and the source text. This scales well but should still be spot-checked by a human on a sample, since automated judges can miss subtle unsupported inferences.
How often should I re-run RAG evaluation after launch?
Re-run the full test set after any change to chunking, embeddings, or prompts. For live traffic, sample faithfulness and citation accuracy weekly, and monitor zero-result queries and citation counts daily through an automated alert, since those signals move faster than a weekly manual review.
Does chunk size affect retrieval metrics more than the embedding model?
In most evaluations, chunk size and splitting strategy affect recall@k more than swapping embedding models, because a chunk that splits an answer across a boundary cannot be retrieved intact regardless of how good the embeddings are. Test chunking changes first before investing in a different embedding model.
Get the Kopik newsletter
New knowledge bases, RAG guides and product news. One email every week or two, unsubscribe in one click.
By subscribing you agree to receive our newsletter. We never share your address.