LLM Evaluation FAQ
How you know an AI system is good — as a number you can fail a release on, not as an opinion from the last demo.
An LLM evaluation system is a golden set of real cases, a rubric for what “good” means, automated scoring, and a CI gate that fails the release when quality drops. Online monitors then sample live traffic so a silent model update shows up on a dashboard, not in a customer complaint.
What evaluation actually is
If you cannot say what would make you roll back a prompt change, you are not ready to ship the change.
What is an LLM evaluation system?
A golden dataset of real cases, a written rubric, automated scoring (including LLM-as-judge where the rubric is stable), traces for every run, and a CI gate. Online monitors then sample live traffic. Together they turn quality into a number you track per release rather than an opinion you debate after a demo.
Why isn’t “it looks good” enough?
Because the next prompt tweak, model swap, or silent provider update will change behaviour in a way nobody reads. Evaluation is how you notice. Teams that skip it do not skip failure — they skip noticing until a customer does.
What is a golden dataset?
A versioned set of real inputs with verified outputs or graded traces, drawn from your actual tickets, documents, or agent runs — not from the internet. It includes cases that must fail (empty retrieval, out-of-policy, must-escalate). If the set is only happy paths, the suite will green-light a system that is dangerous on the long tail.
What is an LLM-as-judge, and when is it trustworthy?
A second model scoring an output against a rubric. It is trustworthy when the rubric is specific, when it is calibrated against human labels on a held-out slice, and when you keep a human on ambiguous cases. It is not trustworthy as the only score, or when the judge is the same model family as the system under test with no check.
How is this different from a model benchmark like MMLU?
Benchmarks compare models on public tasks. Production evals compare versions of your system on your tasks. You can swap a model that wins a leaderboard and lose on refunds, citations, or tool arguments. We use public benches to choose candidates; we use your golden set to choose what ships.
Do we need evals before we have production traffic?
Yes. The first golden set is built from historical tickets, documents, or scripted scenarios — including agent simulations when the path is multi-step. Shadow mode then adds live cases. Waiting for production traffic to “see what happens” is how you let customers be the eval set.
How long does it take to stand up a first eval suite?
A useful first slice — rubric, 50–100 labelled cases, a CI job — lands inside the four-week implementation, usually by week two, because the rest of the build depends on it. Deepening the set is ongoing. We deliberately score a small first batch so you can check the output before volume ramps.
How the suite is built
The work is labelling, rubric writing, and deciding which failures are release-blocking. The tooling is the easy part.
How do you build a golden set from our real cases?
Sample from the last quarter of tickets, chats, or documents, stratified by intent and by the failures you already know about. Your process owner grades against the rubric; we second-review the ambiguous ones. Traces from shadow mode then grow the set. Nothing is labelled in a vacuum — the grades have to match how you already judge a good agent.
How many examples do we need?
Enough to cover the intents you will ship, including must-fail cases, with a held-out slice for calibrating judges. In practice that starts around 50–100 well-labelled cases for a single workflow and grows as live failures are promoted. A thousand unlabelled logs are less useful than eighty graded ones.
How do you score agent tool-use, not just text?
On the trace: which tool, which arguments, whether the side effect was allowed, whether the agent stopped. Text quality and tool correctness are separate scores so a fluent wrong refund does not pass. Function-calling evals include negative cases — tools it must not call.
What is a regression gate in CI?
A job that runs the golden set on every prompt, model, or retrieval change and fails the pipeline if a score drops past a threshold you set. It is the same idea as a unit test. Without it, every “quick prompt fix” is an unreviewed production change.
How do you catch a silent model update?
Online evals on a live traffic sample, plus pinning versions where the provider lets you. When the sample score drops, you get paged. That is also how you notice your own retrieval drift, not only theirs.
Who writes the rubrics?
Your process owner, with us in the room. A rubric nobody in the business will defend is theatre. We bring the structure — dimensions, must-fail cases, inter-rater checks — and we report agreement as a number with every delivery.
How often should we re-label?
When the workflow changes, when a new failure mode shows up more than once, and on a scheduled review so the set does not fossilise. Failures from production are promoted into the golden set as a matter of course. Re-labelling everything quarterly is usually waste; never touching it is how evals go stale.
Evals in production
Online sampling, judge calibration, and ownership after handover are what keep the suite honest once we leave.
How do online evals work on live traffic?
A sample of live traces is scored on the same rubric as the golden set, continuously. It is not a full grade of every request — that is usually too slow and too expensive. The sample is enough to see drift, and full traces stay available when a score drops so you can inspect, not just alert.
What do we do when a score drops?
The release is blocked if it is a CI gate. If it is online, you get paged, inspect the traces, and either pin the previous model, roll back the prompt, or promote the new failures into the golden set. Guessing which of those it is without traces is how teams thrash.
Can evals run inside our VPC?
Yes. The suite runs where the agent runs. Labels, traces, and documents stay in your perimeter. We do not need to take the golden set offsite to score it.
How do you avoid the judge agreeing with itself?
Calibrate against human labels on a held-out slice, use a different model family for the judge where it matters, and keep a human on the ambiguous band. Inter-rater agreement is reported. A judge with no human check will quietly drift to whatever the system started saying.
Who owns the eval suite after handover?
You do. Datasets, rubrics, jobs, and dashboards are yours. We stay on-call for 30 days, then a retainer is optional. Most teams keep a quarterly eval review with us to catch drift; plenty run the suite themselves from day 31.
How is this priced?
As part of the implementation, or as a standalone evaluation service if the symptom is “we cannot tell if it is good.” Pricing is quoted before work starts — per delivered unit against an agreed quality bar, or a fixed monthly fee for a standing programme. You are not billed for batches that fail QA.
Domain and use cases
The suite is the same idea. The rubric is where the domain lives.
How do you evaluate a support agent?
On policy faithfulness, correct escalation, tool-use on the order, and CSAT-risk (tone, loops, invented goodwill). Must-fail cases include identity, legal, and empty retrieval. Deflection is reported beside missed-escalation rate so a quiet customer is not scored as a win.
How do you measure RAG faithfulness?
Claim-level: each sentence in the answer must align to a retrieved span, and citations must point at those spans. Retrieval recall is a separate number so you can tell a miss from a hallucination. Hallucination audits are this discipline as a service on a system you already have.
How do you score a multi-step agent?
On the whole trace, not the final sentence. Process supervision and agent simulations score each step: the plan, the tool calls, the recovery from a failed tool. A fluent summary of a wrong plan is a failure. This is why multi-agent systems need evals before they need more agents.
What does evaluation look like in healthcare?
Must-fail on medical advice, missed escalation, and PHI leakage. Admin intents (scheduling, benefits, intake) are graded against written policy. Missed-escalation rate is a target, not a hope. Humans stay on clinical content. The suite is how you prove that to a review board.
How do you evaluate financial-advice or KYC agents?
On whether the answer stayed inside retrieved policy, whether a required disclosure fired, and whether the agent refused when it should have. Money movement and KYC decisions have human checkpoints; the eval suite scores whether those checkpoints were actually hit. Reconstructable traces are part of the artefact, not a nice-to-have.
LLM evaluation at ReinforcedX is not a leaderboard. It is golden datasets built from your real cases, rubric-driven judges, regression gates in CI, and online sampling of live traffic. Retrieval, tool-use, and safety are scored as separate numbers. The suite ships with the system and is yours at handover. If we cannot define how success is measured, we say so before taking the work.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.