How to Reduce LLM Hallucinations in Production

Key takeaways

• 01

  Most production hallucinations are retrieval misses, stale corpus, or unconstrained generation wearing a fluent sentence — measure those first.

• 02

  Grounded generation means the model may use only retrieved chunks and tool results; prior-knowledge answers are out of policy for enterprise facts.

• 03

  Citations must be span-level and checked after generation; a source list under an unsupported answer is worse than no answer.

• 04

  A golden set of real questions with verified sources, plus a pinned LLM judge, is the only way to know whether a model or prompt change reduced hallucinations.

• 05

  Shadow mode is required before cutover: faithfulness without usefulness is over-refusal, and users will route around a mute agent.

A hallucination is a statement the model presents as fact that is not supported by the evidence you allowed it to use. In a consumer chatbot that is often a wrong date. In production it is a discount that does not exist, a policy clause nobody wrote, a ticket ID that was never opened, or a citation pointing at a document that does not contain the claim. Fluency is not the bug; unsupported confidence is.

Split the category before you try to reduce it. Intrinsic errors contradict the context you already retrieved. Extrinsic errors invent facts outside that context. Citation fabrications look grounded and are not. Empty-retrieval answers are the most common enterprise case: the index missed, the ACL filtered everything, and the model filled the gap from pretraining. Each type has a different fix.

Teams search for how to reduce LLM hallucinations and get temperature folklore. Lower temperature reduces sampling noise; it does not stop a model from stating a plausible policy when retrieval returned nothing. Longer system prompts do the same: the model is still a next-token predictor with no live view of your wiki unless you give it one.

The production playbook is mechanical. Put the right evidence in context (hybrid retrieval, tools, current schemas). Constrain generation to that evidence. Check the output. Measure on questions you already know the answer to. That is grounded generation. See the RAG how-to at /how-to/build-rag-ai-agent-organization-knowledge for corpus and chunking; this guide covers what you add around generation so the model cannot wander.

Grounded generation is a contract: the model may use retrieved chunks and tool results, and it may not use parametric memory for enterprise facts. Implement that contract in three places. The prompt states the rule. The application withholds the user-facing reply if retrieval is empty or the faithfulness judge fails. Tool wrappers return structured errors instead of letting the model invent a payload.

Hybrid retrieval (dense vectors plus BM25) still matters here because enterprise questions are full of IDs and acronyms. Query-time ACLs belong on the same path: filter before generation so restricted text never enters the context window. When filtering empties the result set, treat it like a miss — do not hint that a forbidden document exists.

Users trust answers that look sourced. If you attach three URLs to an unsupported paragraph you have made hallucination harder to spot. Require claim-to-span maps: each factual sentence points at a chunk ID and a character or sentence range. After generation, a checker confirms the span actually contains the claim. Failures become refusals or trimmed answers, never silent extras.

Faithfulness is the fraction of claims supported by the supplied evidence. Groundedness (sometimes used interchangeably) should be scored by a pinned judge model with a written rubric, plus a human sample. Do not let the serving model grade itself. Track citation precision separately — a correct answer with a fake link is still a production defect.

A golden set is a frozen list of real questions, verified answers, and the documents or tool traces that justify them. Start at 80 items if that is all you can verify; 150–200 is a better production floor. Stratify by task (FAQ, policy, multi-hop, tool use) and by risk (wrong tone vs wrong price). Include known-empty cases so refusal is scored, not only answers.

Wire scores into CI the way you wire unit tests. Retrieval hit rate (gold chunk in top 10), faithfulness, citation precision, and useful-answer rate (not refused when evidence exists) are the four numbers that catch most regressions. Pin judge and serving model versions. A silent provider upgrade is a release you did not approve.

Shadow mode runs the grounded path on live questions without showing it to users. Compare old vs new on faithfulness and useful-answer rate. Over-refusal is the usual side effect of a hard grounding rule; you will see it here before customers do. Keep the grounded path dark until both numbers meet the bar you wrote down in week one.

After cutover, every thumbs-down and judge-fail goes to a queue tagged by root cause: missing doc, stale chunk, ACL miss, tool error, or true generation error. Most tickets are corpus or tool tickets. A four-week implementation typically spends week 1 on the error taxonomy and golden set, week 2 on grounding and citations, week 3 on shadow scoring, and week 4 on handover of evals and runbooks — the client owns those artifacts.

• Week 1 — taxonomy, golden set, and current-path baseline scores
• Week 2 — hybrid retrieval or tools, refusal, span citations
• Week 3 — faithfulness judge, CI gate, shadow mode on live traffic
• Week 4 — cutover criteria, runbooks, and 30 days on-call after handover

Step-by-step build

1. 1

   Define hallucination types you will actually score

   Split errors into ungrounded facts, contradictions of retrieved text, fabricated citations, and overconfident answers on empty retrieval. Tag each golden-set item with type and severity so you do not treat a wrong tone as a policy invention.

2. 2

   Close the evidence gap with hybrid retrieval or tools

   For knowledge questions, index the sources that answer real queries with hybrid (vector + BM25) retrieval and query-time ACLs; for live systems, expose read tools instead of hoping the weights know. If the right span is not in context, the model will invent one.

3. 3

   Force grounded generation and refusal

   Instruct the serving model to answer only from retrieved chunks and tool payloads, to cite spans, and to say it does not know when top-k is empty or below a similarity/re-rank threshold. Implement refusal in the application, not only in the prompt, so a jailbroken completion cannot skip it.

4. 4

   Add a post-generation faithfulness check

   Run a second, pinned model (or a rules layer) that scores whether each claim is supported by the supplied evidence. Fail closed on high-severity tasks: strip unsupported sentences or replace the answer with a refusal plus owner.

5. 5

   Build the golden set and CI gate

   Collect 80–200 real questions with verified answers and source docs, stratified by department and failure type. Score retrieval hit rate (right chunk in top 10), faithfulness, citation precision, and useful-answer rate; fail the build if any drops past an agreed threshold.

6. 6

   Shadow, triage, and freeze the loop

   Run the grounded path in shadow mode on live traffic for at least a week, sample thumbs-down and judge-fail traces, and send them to a queue that fixes corpus, tools, or policy. Cut over only when faithfulness and useful-answer rate both hold, then re-score weekly.

Common pitfalls

The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.

Frequently asked questions