RAG FAQ
How retrieval-augmented generation actually grounds an agent — and the failure modes that show up once you leave the demo corpus.
RAG retrieves relevant passages from your own sources at question time and passes only those to the model, with instructions to cite them and to say it does not know when retrieval is empty. It is how you ground answers in current, permissioned knowledge without retraining the model for every document change.
What RAG is — and is not
If the facts change weekly, you need retrieval. If the task is a style or a skill the model already fails at, retrieval will not save you.
What is RAG, in one sentence?
Retrieval-augmented generation fetches relevant passages from your own sources at question time and asks the model to answer only from those passages, with citations. The model is not required to memorise your wiki; it is required to use what was retrieved, and to refuse when nothing relevant was found.
How is RAG different from fine-tuning?
Fine-tuning changes the model’s behaviour. RAG changes the evidence in the prompt. Use RAG when the source of truth is documents that update; use fine-tuning when the model must acquire a skill, a format, or a voice that prompting cannot hold. Many production systems use both: RAG for facts, a light fine-tune or a tightly written system prompt for tone.
What sources can we ground an agent on?
Anything you can export or query: wikis, Google Docs, SharePoint, PDFs, tickets, CRM notes, databases, and HTML. The constraint is not format, it is access control and quality. Start with two or three curated sources that actually answer the questions people ask, then expand. Indexing the whole drive on day one fills the index with drafts and duplicates.
Why do RAG systems still hallucinate?
Usually because retrieval returned the wrong passage, the prompt did not forbid ungrounded claims, or the model was allowed to answer when the index was empty. Hallucination audits catch claim-level grounding failures against the source. The fix is better retrieval, a refuse-when-empty policy, and a citation check — not a larger model.
How do you know retrieval is working?
A golden set of real questions with verified answers and known source passages. You measure recall of the right chunk, faithfulness of the answer to that chunk, and citation correctness as separate numbers. If you only score “the answer looks good,” you cannot tell a retrieval failure from a generation failure.
Do we have to put all our documents in one place first?
No. Connectors pull from the systems of record and keep an index in sync. You do need an owner for each source and a decision on what is in-scope. A messy corpus with no owner will produce messy answers regardless of the retriever.
How long does a first RAG agent take?
A working prototype can be up in days on one or two folders. A production agent — permissions, hybrid search, evals, incremental sync — sits inside the four-week implementation. Most teams have it answering real questions in shadow mode by week three.
How a RAG agent is built
Chunking, hybrid retrieval, and ACLs are the parts that decide whether the system is useful or a liability.
How should documents be chunked?
Along heading boundaries, typically 300–800 tokens, with the document title and heading path prepended to every chunk. That single prefix — “Pricing Playbook → Enterprise tier → Discounts” — usually lifts answer accuracy by double digits. Split too small and you lose context; split too large and retrieval returns noise.
Vector search or keyword search?
Both. Embeddings catch paraphrase; BM25 catches the SKU, the error code, the policy number. Production retrieval is hybrid, then a cross-encoder re-ranker on the top candidates. Teams that ship vector-only spend the next quarter wondering why identifier queries fail.
How do permissions work at retrieval time?
The asking user’s identity is resolved against your IdP, and the retriever filters to documents that user can already open. A single service account over the corpus is simpler and leaks. In practice: pilot on one shared folder, then switch to per-user OAuth before expanding org-wide.
What is agentic RAG?
The model is allowed to retrieve more than once — rewrite the query, follow a citation, fetch a related doc — instead of a single top-k dump into the prompt. It helps on multi-hop questions. It also multiplies cost and failure modes, so it sits behind evals and a step limit, not as the default for every FAQ.
How do you keep the index fresh?
Incremental sync from the source’s change feed, not a nightly full crawl. On Drive that is changes.list; elsewhere, webhooks or a short poll. Stale indexes are how last week’s pricing becomes this week’s answer. Freshness is an SLO you can measure: minutes from edit to searchable chunk.
What model do you use for embeddings?
Whatever you already run that is good enough on a held-out retrieval set — we do not lock you to a vendor. The embedding model is swapped less often than the generator. What matters more is consistent chunking, metadata, and a re-ranker. We measure recall on your questions before arguing about embedding brands.
How much data do we need to start?
Enough to answer the first workflow’s questions, plus 50+ of those questions with verified answers for the golden set. That is usually two or three curated sources, not the company-wide dump. Volume without labelled questions is how teams spend a month indexing and still cannot tell if it works.
Production RAG
Faithfulness, refuse-when-empty, and citations are the production bar. Everything else is how you get there.
How do you evaluate RAG quality?
Three scores, kept separate: retrieval recall (did we fetch the right passage), faithfulness (does the answer stay inside that passage), and citation correctness. Rubric judges can score faithfulness; they cannot replace a golden set of real questions. Regression gates in CI fail the release if any of the three drops.
What happens when retrieval returns nothing?
The agent says it does not know and offers a handoff. It does not guess. That behaviour is tested as a first-class eval case, including questions that look in-domain but have no source. Ungrounded answers are blocked before they ship.
Can we cite sources in every answer?
Yes, and you should. Each answer carries the document URL and the heading path of the chunks used. Citations make the agent a navigator, which is how teams actually trust it, and they give you a handle for faithfulness checks.
Where does the index live?
In your cloud, next to the agent. We do not copy the corpus onto ReinforcedX infrastructure, and the contents are not used to train shared models. You choose the vector store — managed or self-hosted — and the credentials stay in your secret manager.
How do you stop the model from ignoring retrieved context?
A grounded system prompt, structured outputs where the schema requires a citation, and a claim-verification pass on anything customer-facing. If a claim cannot be aligned to a retrieved span, it is stripped or the answer is refused. This is cheaper than hoping the model “usually” listens.
What usually goes wrong in the first 90 days?
Indexing everything, one service account for every user, no eval set, and no refuse-when-empty path. Those four show up in almost every failed RAG rollout we are asked to rescue. The how-to guide and the evaluation FAQ cover the antidotes in more depth.
Domain and use cases
Same retriever, different corpus, different definition of a wrong answer.
How does RAG work over legal and policy documents?
Heading-aware chunking, versioned sources, and a refusal when the retrieved policy is silent. Citations are mandatory because a lawyer will open the link. Updates must land in the index within minutes of a policy publish. We do not let the model reconcile two conflicting policies on its own — that routes to a person.
Can RAG power a support knowledge base?
Yes. It is one of the highest-ROI first agents: grounded answers on order policy, with citations, inside the helpdesk or on chat, escalating when retrieval is weak. Pair it with the customer support FAQ — RAG is how the agent knows; the support workflow is how it acts.
How do you ground HR and employee-policy questions?
On the current handbook, benefits docs, and country-specific addenda, with per-user filtering so an employee in one region does not retrieve another’s policy. Answers cite the section. Anything that affects pay, leave, or discipline and is not explicitly in the retrieved text is escalated, not inferred.
What about engineering wikis and tickets?
Wikis plus a ticket index, with title and heading prefixes, work well for “where is X documented?” They fail when the wiki is stale — which is why incremental sync and a freshness SLO matter. Pairing with Slack, as in the how-to, is how most internal agents actually get used.
Can RAG run over regulated research or PHI?
Yes, inside your perimeter, with zero-retention provider settings and per-user permissions at retrieval time. PHI and unpublished research never leave your tenancy. The model is not trained on that corpus. If a use case would require sending raw documents to a provider that cannot meet that bar, we say so and do not do it.
Retrieval-augmented generation (RAG) is the pattern of chunking your documents, retrieving the most relevant passages, and generating an answer only from that context. Fine-tuning changes how a model talks; RAG changes what it is allowed to know today. Production RAG needs hybrid search, permission filtering at retrieval time, citations, an eval set of real questions with verified answers, and a defined behaviour when retrieval is empty. ReinforcedX builds RAG agents inside your perimeter and hands over the index, evals, and runbook.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.