LLM Evaluation Systems (Evals)
How to build production LLM evaluation systems: golden datasets, rubric-based LLM judges, regression gates in CI, online monitoring, and the eval-driven development loop.
- Read time
- 15 min
- Level
- Intermediate
- Updated
- 2026-05-22
- Components
- Golden dataset · Rubric judges · CI regression gate
An LLM evaluation system measures AI quality continuously using a golden dataset of real cases with expected outcomes, automated graders (exact checks plus rubric-driven LLM judges), and a CI gate that blocks prompt or model changes that regress scores — extended with online monitoring that samples and grades live production traffic.
Key takeaways
- 01
Teams without evals discover regressions from angry users; teams with evals discover them in CI before deploy.
- 02
A golden set of 100–300 real, stratified cases beats thousands of synthetic ones — production traces are the best source.
- 03
LLM-as-judge grading is reliable when given a binary-criteria rubric, not a 1–10 vibe scale, and is calibrated against human labels.
- 04
Every prompt, model, or retrieval change should run the eval suite in CI exactly like a test suite — eval-driven development is the LLM analog of TDD.
- 05
Online evals (sampling and grading live traffic) catch drift that offline suites miss: data drift, upstream model updates, and new user behaviors.
Why evals are the control system for AI products
Every LLM feature degrades invisibly. A prompt tweak that fixes one case breaks three others; a model version bump shifts tone; a retrieval change silently drops a document type. Without measurement, each change is a gamble, and teams either freeze (shipping nothing) or churn (shipping regressions). An evaluation system converts AI quality from anecdote to number, which is what makes iteration safe and fast.
The practical payoff is velocity. Teams with solid evals ship prompt and model changes daily because a green suite de-risks the deploy; teams without them re-test manually, argue from single examples, and slow to a crawl. Evals are not QA overhead — they are the mechanism that lets you move.
Golden datasets: small, real, stratified
The foundation is a golden set: real inputs with verified expected outcomes. Quality dominates quantity — 100 to 300 well-chosen cases catch the vast majority of regressions if they are stratified across intents, difficulty tiers, and known failure modes (ambiguous queries, adversarial inputs, empty-retrieval cases, multilingual inputs). Harvest them from production traces, not imagination; the best eval case is yesterday's real failure, labeled and frozen.
Treat the golden set like code: versioned, reviewed, and grown deliberately. Every production incident should end with "add the case to the suite". Refresh stale cases quarterly — a golden set that no longer resembles live traffic gives green lights to broken systems.
Graders: deterministic checks plus rubric judges
Use the cheapest grader that can judge each property. Deterministic checks — exact match, regex, JSON-schema validation, citation-presence, latency and cost thresholds — are free, instant, and unambiguous; route every property you can to them. What remains (helpfulness, groundedness, tone, reasoning quality) goes to an LLM judge.
LLM judges are reliable under three disciplines. Give them binary criteria ("Does the answer cite a retrieved source? yes/no"), not 1–10 scales — decomposed binary rubrics agree with humans far better than holistic scores. Calibrate the judge against a few hundred human-labeled examples and report its agreement rate. And freeze the judge model/prompt version; an unversioned judge is a moving goalpost.
- Deterministic: schema validity, exact/contains, citation presence, latency, cost
- Judge rubric: decomposed binary criteria, with reasons logged per verdict
- Calibration: judge vs. human agreement measured before trusting the judge
- Pairwise mode: for subjective quality, judge A-vs-B comparisons instead of absolute scores
The regression gate: evals in CI
Wire the suite into CI so any change to prompts, models, retrieval, or tools runs the golden set automatically and posts a scorecard: overall pass rate, per-stratum deltas, cost and latency deltas, plus a diff of newly failing cases. Block merge on hard-criterion failures (schema validity, safety, groundedness) and require explicit human sign-off for soft-criterion drops.
Per-stratum reporting is what makes the gate useful rather than noisy — an aggregate score that drops 1% is ignorable; "multilingual queries dropped 12%" is actionable. Keep suite runtime under ~10 minutes by parallelizing calls and reserving the judge for cases that pass deterministic checks.
Online evaluation: grading production
Offline suites test what you anticipated; production tests what you didn't. Sample a percentage of live traffic (start at 5–10%), run the same rubric judges asynchronously over the traces, and dashboard the scores alongside operational metrics. Alert on trend breaks — a groundedness score sagging over a week usually means corpus drift or an upstream model update arrived unannounced.
Close the loop both ways: low-scoring production traces flow into a triage queue, and triaged failures become new golden-set cases. This pipeline — production failure → labeled case → CI gate — is the compounding asset; six months in, your eval suite encodes everything your product has ever gotten wrong.
A maturity path that works
Start embarrassingly small: 30 cases in a spreadsheet and a script that runs them on every prompt change beats a grand platform plan that ships next quarter. Then grow in order: deterministic graders → a calibrated judge → CI gating → per-stratum dashboards → online sampling → automated triage. Each stage pays for the next. The teams that win are rarely the ones with the fanciest eval infra — they are the ones who never ship a change without running the suite.