RAG Architecture
Enterprise RAG architecture: ingest, hybrid index, retrieve, grounded generation, evals, SLOs, and when retrieval-augmented generation is the wrong design.
- Read time
- 16 min
- Level
- Intermediate
- Updated
- 2026-08-25
- Components
- Ingest pipeline · Hybrid index · Retriever
A RAG architecture is a production pipeline that ingests an organization's sources into a hybrid index, retrieves permission-filtered passages at question time, and generates an answer only from that evidence — with citations, a refuse-when-empty policy, and evals that score retrieval hit rate and faithfulness.
Key takeaways
- 01
Retrieval quality dominates model choice: the generator cannot recover from the wrong passages, a stale index, or an empty result set.
- 02
Treat ingest, index, retrieve, generate, and evaluate as five separately owned subsystems, each with an SLO, owner, and failure mode.
- 03
Single-shot RAG is the default enterprise design; add iterative retrieval only when queries are multi-hop or underspecified.
- 04
Do not use RAG when the problem is a skill, a format, or a deterministic computation — retrieval will not fix those.
- 05
Most production RAG incidents come from index lag, ungrounded answers on empty retrieval, and unmeasured hit rate — not from the LLM itself.
Retrieval-augmented generation architecture: five stages
RAG architecture, also called retrieval-augmented generation architecture, is the system that keeps answers tied to current, owned documents instead of model memory. Enterprise RAG design splits into five stages that must be independently operable: ingest (extract, normalize, attach metadata), index (hybrid lexical plus dense), retrieve (query, filter, re-rank), generate (answer only from retrieved evidence), and evaluate (hit rate, faithfulness, latency, cost).
The architecture is a control path, not a prompt. Indexes version, retrievers enforce ACLs, generators refuse without sources, and evals gate deploys. If those contracts live only in a notebook, you have a demo, not a system.
- Ingest: structure-aware extraction, metadata, incremental refresh
- Index: keyword plus embeddings over the same chunk identity
- Retrieve: query rewrite, ACL filter, fusion, re-rank
- Generate and evaluate: citations, refusal, golden-set scores
Ingest, freshness, and corpus control
Ingest is the reliability layer. Every document needs a stable ID, source URL, heading path, owner, modified time, and ACL pointer so later stages can filter and cite. Incremental refresh on change events beats nightly full rebuilds; the SLO that matters is index lag — how long a published policy can remain invisible to retrieval.
Corpus control is architectural, not editorial afterthought. Duplicate drafts, expired pages, and comment threads poison ranking. A production ingest path has an allow-list of sources, a tombstone for deletes, and a quarantine for parse failures so a bad PDF does not silently drop a policy from the index.
Enterprise RAG design of the retriever
The retriever is a subsystem with its own contract: given a user, a query, and a latency budget, return k passages the user is allowed to see, ranked for answerability. Hybrid retrieval (BM25 plus embeddings, fused, then re-ranked) is the production default because enterprise queries mix natural language with IDs, error codes, and product names.
Permission filtering belongs inside the retriever, before generation, against live entitlements — not baked into the index and not applied after the model has already seen restricted text. Keep the retriever callable as an API so the same index can serve a chatbot, an agent tool, and an offline eval job without forked ranking logic.
Grounded generation and refusal
The generator's job is narrower than "be helpful": produce an answer whose claims are supported by the retrieved set, attach citations (document URL plus heading path), and refuse when retrieval is empty or off-topic. Stuffing twenty weakly related chunks is not grounding; it is context noise that invites the model to pick a plausible sentence.
Prompt contracts should be versioned with the retriever, not edited in a chat UI. Freeze the system instruction that forbids ungrounded claims, and treat citation presence as a deterministic grader. If a claim cannot be pointed at a span, it does not ship.
Failure modes and when RAG is the wrong design
Classic RAG fails in predictable ways: the right document was never ingested; the chunk split the table from its header; lexical search missed a paraphrase; dense search missed an exact SKU; the model answered anyway. Debugging generation while retrieval hit rate is unmeasured is the usual trap — most "hallucinations" are misses wearing a fluent sentence.
Do not choose RAG when the task is style, tool use, or exact computation. A model that cannot write SQL will not learn it from retrieving schema docs; a refund amount should come from a ledger API, not a prose chunk. Tiny, closed corpora that fit in the prompt also do not need an index. RAG is for knowledge that is large, changing, and permissioned.
- Stale or duplicate corpus: ranking surfaces the wrong version
- Empty retrieval answered anyway: fluent, ungrounded policy
- Context stuffing: too many weak chunks, citations that do not support claims
- Wrong problem class: skill, format, or transactional truth mistaken for search
Evaluating and operating RAG in production
Operate RAG like any other search-plus-generation system. Offline: a golden set of real questions with verified answers, split into retrieval metrics (recall@k, hit rate) and generation metrics (faithfulness, citation validity, refusal on empty). Online: index-lag dashboards, p95 retrieval latency, cost per answer, and sampled judges on live traffic. Gate prompt, embedding, and ranking changes in CI the same way you gate code — a drop in hit rate on one stratum (acronyms, multilingual, new product line) is the signal, not a 1% average.