How to Evaluate a RAG System
How to evaluate a RAG system: golden sets, retrieval hit rate, faithfulness score, RAGAS-style metrics, LLM judges, and a CI gate that catches regressions.
To evaluate a RAG system, freeze a golden set of real questions with known source chunks, score retrieval (hit rate, MRR) separately from generation (faithfulness, answer correctness), and fail CI when those numbers drop on a pinned pipeline. RAGAS-style metrics are useful once they run on your traces; they are not a substitute for gold documents and a human sample.
What you’ll build
- A stratified golden set the client owns, with gold documents and acceptable answer notes
- Separate retrieval metrics (hit rate, MRR, nDCG) and generation metrics (faithfulness, answer correctness)
- A RAGAS-style or equivalent harness wired to traces, not to a notebook that ran once
- A CI gate on retrieval hit rate and faithfulness, with judge version pinned
- A weekly slice of live traffic graded in shadow mode so drift shows up before users file tickets
Before you start
- 01A frozen RAG pipeline (chunker, embedding model, hybrid retriever, generator) you can pin by version
- 02Access to production or shadow traces: query, retrieved IDs, answer, citations
- 03At least 80 real questions with verified answers and source chunk or doc IDs
- 04A judge model endpoint separate from the serving generator, with a pinned revision
- 05CI that can run a batch eval job and fail the build on metric drop
Key takeaways
- 01
Retrieval hit rate (gold chunk in top-k) is the first number; if it is low, do not debug the generator.
- 02
Faithfulness scores whether claims are supported by retrieved text; answer correctness scores whether the user got the right fact — they can diverge.
- 03
Hybrid retrieval (vector + BM25 + re-ranker) must be in the eval path you ship, not only in a research notebook.
- 04
Pin chunker, embeddings, retriever, prompt, generator, and judge; any unpinned piece makes week-to-week scores incomparable.
- 05
Shadow-mode samples of live traffic catch query drift that a static golden set will miss.
Why RAG eval splits retrieval from generation
A RAG system is two products glued together. Retrieval must put the right span in the window. Generation must use that span and not invent the rest. If you only score “did the answer look good,” you cannot tell which product failed. That is why “how to evaluate RAG” starts with hit rate, not with a chat rubric.
Typical pattern: hit@10 is 54%, faithfulness on questions that hit is 91%, and overall answer correctness is 61%. The work is in the index, not in a warmer system prompt. The reverse pattern — high hit rate, low faithfulness — is unconstrained generation or a prompt that allows prior knowledge. Split the metrics or you will apply the wrong fix.
Golden sets that look like production
Invented FAQs under-test acronyms, SKUs, and “compare last quarter’s policy to the draft.” Mine queries from actual behavior, then have owners verify the source and the answer. Include empty-gold items: questions you should refuse, and questions that should return empty after ACL filtering. If empty cases are missing, a chatty model looks like a quality win.
Stratify by source (wiki, tickets, PDFs), intent (lookup, compare, how-to), and risk (wrong cafeteria hours vs wrong discount). Size: 80 is a start; 150–300 is a working production set. Version the labels. When a document moves, update gold IDs; do not let a moved heading look like a retriever regression.
Retrieval hit rate, hybrid search, and permissions
Hit@k is 1 if any gold chunk appears in the top k, else 0. Report k=5 and k=10. MRR records where the first gold sat. nDCG is useful when several chunks are valid. Always compare three systems on the same set: vector-only, hybrid (vector + BM25 with rank fusion), and hybrid plus a cross-encoder re-ranker. Enterprise queries are ID-heavy; hybrid usually wins recall.
Query-time ACLs belong in the metric. If the retriever returns a gold chunk the user cannot read, count a leak, not a hit. Existence leakage — empty-for-permission looking different from empty-for-miss — should also be tested. Eval users should include at least two roles.
Faithfulness, answer correctness, and RAGAS metrics
Faithfulness: are generated claims supported by the retrieved context? Answer correctness: does the answer match the gold fact even if phrasing differs? Citation precision: do cited spans contain the claim? These three catch the failures users report. A model can be faithful to the wrong chunk (low correctness) or correct by guessing without using context (low faithfulness). You want both high.
RAGAS-style metrics (context precision, context recall, faithfulness, answer relevancy) are a reasonable default library if they run on your traces and your gold. They are not magic: they depend on a judge model you must pin, and they drift when that model changes. Keep a human-labeled subset as the calibration stick. Never let a library’s default judge float to “latest.”
LLM judges and the CI gate
Judges need a rubric, not “score 1–5.” Spell out what a 0 faithfulness looks like (unsupported number, fabricated citation). Run the judge on the same context the generator saw. Dual-score a sample with two humans until agreement is acceptable; then use humans as a weekly audit, not as the full batch.
CI is the difference between a demo metric and a system. Pin chunker, embedding model, index snapshot or replay, prompt, generator, judge. Fail the pipeline on hit@10 or faithfulness drop beyond the bound. A four-week build typically lands the golden set and harness in weeks 1–2, baselines in week 3 (shadow), and the gate plus runbooks in week 4. The client owns the eval suite.
- Gold: real questions, chunk IDs, empty/ACL cases, strata
- Retrieval: hit@5/10, MRR, hybrid vs vector, permission leaks
- Generation: faithfulness, correctness, citation precision
- Process: pinned judge, CI fail, weekly live sample
What eval setups fail to catch
Static golden sets miss new product names. Judges miss numerically close errors (“3 days” vs “5 days”) if the rubric is vague. Average scores hide one toxic stratum. Latency and cost are not quality, but a re-ranker that triples p95 may be rejected even if nDCG ticks up — measure them on the same run.
Do not stop at offline eval. Shadow mode on live traffic, with traces sampled into the same judge, is how you see query drift. When faithfulness holds and thumbs-down rise, you are usually over-refusing or citing unreadable sources. That is still an eval finding.
Step-by-step build
- 1
Build a stratified golden set
Pull 80–200 real questions from search logs, tickets, and interviews. For each, record the gold document or chunk IDs, a verified answer, intent tag, and whether retrieval should be empty (permission or unknown). Hold the set in version control; the client owns it.
- 2
Instrument traces
For every eval run, store query, user role, retrieved IDs and scores, hybrid vs rerank ranks, prompt hash, answer, and citations. Redact PII. You cannot compute hit rate without IDs, and you cannot debug faithfulness without the chunks the model actually saw.
- 3
Score retrieval independently
Compute hit@k (gold chunk in top 5 and top 10), MRR, and nDCG. Break out vector-only vs hybrid vs post-rerank so you know which stage failed. Measure permissioned retrieval: a hit that violates the asker’s ACL is a failure, not a win.
- 4
Score generation: faithfulness and correctness
Run a pinned judge with a rubric: each claim supported by retrieved text (faithfulness), answer matches gold (correctness), citations point at real spans, refusal when gold is empty. Sample 10–20% for humans. RAGAS-style context precision/recall can sit beside these, not instead of them.
- 5
Put the harness in CI
On every change to chunking, embeddings, index, prompt, or model, run the golden set. Fail if hit@10 or faithfulness drops more than the agreed delta (for example 3 points) versus the last pin. Publish a per-stratum report, not one average.
- 6
Add live shadow scoring
Each week, grade a sample of production queries: retrieval coverage (did we return anything useful), faithfulness, thumbs-down root cause. Feed misses into corpus tickets. Re-gold new question types so the static set does not rot.
Common pitfalls
The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.
Scoring only the final answer
A fluent wrong answer often means the right chunk never made top-k. Split retrieval hit rate from faithfulness or you will spend weeks rewriting prompts for an index problem.
Letting the serving model grade itself
Self-scoring inflates faithfulness. Use a pinned judge, a written rubric, and a human sample. When the judge model upgrades, re-baseline — do not compare new scores to old ones.
A golden set of invented FAQs
Synthetic questions miss acronyms, typos, and the two-hop questions employees actually ask. Seed from tickets, search logs, and support tags, then verify answers with owners.
One aggregate score for the whole corpus
HR policy can look healthy while engineering runbooks fail. Stratify by source, intent, and permission class or you will ship a dashboard that cannot be acted on.
No CI gate, only a slide
Embedding model upgrades, chunker tweaks, and prompt edits all move quality. If eval is optional, it will be skipped the week you need it. Fail the build when hit rate or faithfulness drops past the bound.