How to Evaluate a RAG System: Retrieval Metrics, Faithfulness and Citation Accuracy

Evaluating a RAG system means measuring two separate things: whether retrieval surfaces the right passages, and whether the generated answer faithfully reflects them. The most reliable approach combines retrieval metrics such as recall@k and MRR, faithfulness and citation checks against a test set of real questions, and ongoing monitoring once the system is live. Skip any one of these and you will ship a demo that looks impressive but fails quietly in production.

Why a RAG system needs its own evaluation approach

A retrieval-augmented generation pipeline has two distinct failure points, and they need two distinct sets of metrics. Retrieval can fail by missing the right passage, burying it below irrelevant results, or returning stale content. Generation can fail even when retrieval succeeds, by ignoring the retrieved text, blending it with the model's own prior knowledge, or misattributing a claim to the wrong source. If you only score the final answer, you cannot tell which layer broke, and you will waste time tuning the wrong component. How RAG works, step by step is worth reading first if you want a mental model of where each metric below actually applies in the pipeline.

This is also why evaluating RAG is harder than evaluating a standalone chatbot. A fluent, confident answer can still be wrong, and a technically correct answer can still cite the wrong paragraph. Good evaluation treats retrieval quality, faithfulness and citation accuracy as three separate scorecards rather than one pass or fail judgement.

Retrieval metrics: recall@k, precision@k and MRR

Before you worry about what the model says, check what it was given. Retrieval metrics compare the passages your system actually returns against a gold set of passages a human has marked as relevant for each question.

The three metrics worth tracking

• Recall@k: of all the relevant passages that exist in the knowledge base, what proportion appear somewhere in the top k results? This is the metric that tells you whether the answer was even available to the model.
• Precision@k: of the k passages returned, what proportion are actually relevant? Low precision means the model is wading through noise, which increases the chance it latches onto the wrong sentence.
• Mean Reciprocal Rank (MRR): for each query, take the position of the first relevant passage (1st, 2nd, 3rd...) and score it as 1 divided by that position, then average across all queries. MRR rewards systems that put the right passage near the top, not just somewhere in the list.

In practice, most recall problems trace back to one of two causes: the chunking strategy split a relevant fact across two chunks so neither scores well on its own, or the retriever relies purely on vector similarity and misses an exact term match such as a product code or a statute number. Chunking Strategies for RAG and Hybrid Search: Why BM25 Plus Vectors Beats Either Alone for RAG both cover fixes for this, and it is worth re-running your recall@k numbers after any change to either.

Answer faithfulness: does the answer stay grounded in the sources

Faithfulness asks a narrower question than correctness: given the passages that were retrieved, does the answer only state things those passages support? An answer can be faithful but still wrong, if the retrieved passages themselves were outdated or incomplete. It can also be unfaithful while happening to be factually correct, if the model filled a gap from its own training data rather than the document. For most regulated or high-stakes use cases, faithfulness matters more than raw correctness, because it is what determines whether you can trust and audit the answer.

There are three practical ways to measure it. Human review against a checklist is the gold standard but slow and expensive at scale. Natural language inference style checking, where you test whether each sentence in the answer is entailed by the retrieved passages, can be automated but needs careful calibration. The most common middle ground is an LLM-as-judge: a separate prompt that is shown the question, the retrieved passages and the generated answer, and asked to flag any claim not supported by the passages. Reducing AI Hallucinations: A Practical Guide to Grounded Answers goes into more depth on the mechanisms behind ungrounded answers and how to reduce them at the retrieval and prompting stage, before you even get to measurement.

Citation accuracy: checking the passages, not just the prose

If your system cites sources, the citations themselves need their own evaluation, separate from the prose. A citation can fail in several distinct ways: it points to a real passage that does not actually support the claim next to it, it points to the right document but the wrong section, the quoted text has been paraphrased so heavily it no longer matches the source wording, or the citation is simply missing for a claim that needed one. Document Q&A With Citations: How to Check an AI Answer walks through a manual checklist for exactly this, and the same checklist scales into an automated test once you have a test set.

Two metrics are worth tracking here: citation precision (of the citations given, how many actually support the adjacent claim) and citation recall (of the claims that need support, how many actually have a citation attached). Systems that answer in raw passages mode, returning the retrieved text verbatim rather than a generated synthesis, sidestep most of this risk because there is no paraphrasing step to introduce drift, which is one reason it is worth offering both modes rather than forcing every query through generation.

Building a test set of real questions

Synthetic questions generated by an LLM tend to be too clean: well-formed, single-topic, and phrased the way the documents themselves are phrased. Real users ask messier questions, combine two topics in one query, use the wrong terminology, or ask something the knowledge base only partially covers. A test set built only from synthetic questions will overstate how well your system performs.

1. Mine real questions from support tickets, search logs or sales call notes, aiming for at least 50-100 to start and growing over time.
2. For each question, manually identify the gold passage(s) that should be retrieved and write a reference answer.
3. Deliberately include edge cases: questions with no good answer in the knowledge base, ambiguous questions, and questions that require combining two separate documents.
4. Tag each question by difficulty and topic so you can see where performance drops, rather than relying on a single blended score.
5. Version the test set alongside your knowledge base content, so a score drop can be traced to either a content change or a pipeline change, not both at once.
6. Re-run the full set after any change to chunking, the embedding model, or the retrieval configuration, not just before a big release.

If you are building the knowledge base itself with a service like Kopik, it helps that answers already come back with cited passages rather than opaque prose. That makes it far easier to build a gold-standard test set in the first place, since you can compare the system's cited passage directly against the one your reviewer picked, rather than trying to infer which part of a long document the answer was drawing from.

Continuous monitoring once the system is live

A test set catches regressions before release, but production traffic will always surface questions you did not anticipate. Monitoring closes that gap, and it does not need to be elaborate to be useful.

• Sample a percentage of live queries each week and run them through the same retrieval and faithfulness checks used in testing.
• Track retrieval metrics over time, not just at launch, since reindexing, source document updates or new uploads can quietly shift recall@k.
• Flag low-confidence or no-citation answers for human review rather than letting them pass silently to the user.
• Add a lightweight feedback signal, such as a thumbs up or down next to each answer, and route negative feedback into your test set.
• Re-evaluate whenever source documents change meaningfully, since a knowledge base that grows by 30% can change which passages compete for the top retrieval slots.

If your monitoring involves logging user queries, treat that as personal data handling in its own right rather than an afterthought. UK GDPR and RAG: What to Check Before You Build a Knowledge Base sets out what to document before you start retaining query logs for evaluation purposes.

Teams building their own retrieval pipeline from scratch often find it faster to validate the evaluation approach against an existing service first. The developer documentation covers how answers, cited passages and raw passages mode are returned via the API, which is useful as a reference implementation even if you end up building your own retrieval stack.