How to Build an AI Agent That Follows Your Organization's RAG

Key takeaways

• 01

  RAG quality is determined more by corpus hygiene and chunking than by model choice — a better model cannot compensate for retrieving the wrong text.

• 02

  Hybrid retrieval (dense vectors + BM25 keyword) with a re-ranker outperforms either method alone on enterprise queries, especially for IDs, acronyms, and product names.

• 03

  Permission filtering must happen at query time against live ACLs; baking permissions into the index goes stale the moment access changes.

• 04

  Agentic RAG — letting the model decide when to search, reformulate queries, and iterate — beats single-shot retrieval on multi-hop questions by a wide margin.

• 05

  An enterprise agent must be instructed to answer only from retrieved context and to say "I don't know" on empty retrievals; this single rule eliminates most hallucinated policy answers.

A general-purpose LLM knows the world; it knows nothing about your travel policy, your deployment runbook, or which discount tiers sales can offer. Retrieval-augmented generation closes that gap by fetching relevant internal content at question time and instructing the model to answer from it — turning the model from a confident generalist into a grounded specialist that cites sources.

The "agent" framing adds one more layer: instead of a fixed retrieve-then-answer pipeline, the model itself decides when to search, what to search for, and whether the results are sufficient — looping until it can answer or honestly conclude that it cannot.

Most failed RAG deployments fail at the corpus, not the model. Indexing everything indiscriminately fills the store with duplicates, drafts, and obsolete policies — and retrieval faithfully surfaces that garbage. Start narrow: pick the two or three sources that answer the most actual questions (usually the HR policy space and the engineering wiki), and expand only after quality is proven.

Treat metadata as a first-class citizen. Every chunk should carry its source URL, document owner, last-modified date, and access-control list. Freshness metadata enables recency boosts; ownership enables "ask the owner" escalation; ACLs enable permission filtering. Retrofitting metadata after indexing is far more painful than capturing it on ingest.

Chunking is the highest-leverage tuning knob in the pipeline. Fixed-size splitting tears tables from their headers and steps from their procedures; structure-aware splitting along headings and paragraphs keeps semantic units intact. Aim for 300–800 tokens with 10–15% overlap, and always prepend the document title and heading path to the chunk text — the embedding then encodes context that the raw chunk lacks.

For embeddings, modern general-purpose models are sufficient for most corpora; domain fine-tuning matters less than getting chunking and hybrid retrieval right. Re-embed the corpus when you change models — mixed-model indexes silently degrade similarity.

Dense vector search excels at paraphrase ("how much PTO do I get" → vacation policy) but is weak on exact tokens — error codes, SKUs, acronyms, people's names. Keyword search (BM25) has exactly the opposite profile. Enterprise queries are full of exact tokens, so run both and merge with reciprocal rank fusion; then apply a cross-encoder re-ranker over the merged top ~50 to produce the final 5–10 chunks.

This three-stage pattern — broad hybrid recall, precise re-rank — is the de facto standard because it is cheap where volume is high and accurate where it counts. Measure it with retrieval hit rate against a golden set before ever debugging the generation side: if the right chunk is not in the top 10, no prompt will save the answer.

The fastest way to get a RAG agent banned is answering a salary-band question from a restricted HR doc. Permission filtering must be enforced server-side at query time: resolve the asking user's identity, fetch their live group memberships, and filter candidate chunks against each document's ACL before anything reaches the model.

Two non-negotiables. First, filter before generation, not after — a model that has seen restricted text can leak it in paraphrase. Second, make empty-after-filtering indistinguishable from genuinely-empty retrieval, so the agent never reveals that a restricted document exists.

Single-shot retrieval handles single-hop questions. Real questions are messier: "compare our parental leave to the new policy draft" needs two searches; "why did checkout latency spike last week" needs a search, a follow-up, and synthesis. Agentic RAG exposes retrieval as a tool: the model writes its own queries, inspects results, reformulates, and searches again until it has enough to answer.

Constrain the loop — maximum 3–5 retrieval calls, a token budget, and required citations on every factual claim. Add a final groundedness instruction: if the answer is not supported by retrieved chunks, say so and name the document owner to ask. An agent that reliably says "I don't know, ask the policy owner" earns more trust than one that always answers.

Build a golden set of at least 100 real questions with verified answers and source documents, stratified across departments. Track three numbers weekly: retrieval hit rate (right chunk in top 10), answer groundedness (claims supported by retrieved text), and deflection (questions resolved without human escalation). Wire user feedback into a triage queue — and notice that most thumbs-down answers trace to corpus problems (stale doc, missing doc, duplicate) rather than model problems. The loop that fixes documents, not prompts, is the loop that compounds.

Step-by-step build

1. 1

   Inventory and prioritize the corpus

   Catalog knowledge sources (wikis, drives, tickets, policies), rank them by query value and freshness, and start with the two or three sources that answer the most real employee questions.

2. 2

   Build the ingestion pipeline

   Extract clean text per format, preserve heading structure and metadata (owner, date, ACL, source URL), and re-ingest incrementally on document change events.

3. 3

   Chunk with structure awareness

   Split along heading and paragraph boundaries into 300–800 token chunks with modest overlap, prepending each chunk with its document title and heading path.

4. 4

   Index with hybrid retrieval

   Embed chunks into a vector store and index the same chunks in a keyword engine; at query time run both, merge with reciprocal rank fusion, and re-rank the top ~50 down to 5–10.

5. 5

   Wire the agent with retrieval as a tool

   Expose search as a tool the LLM calls with self-written queries, allow iterative re-querying, enforce permission filters server-side, and require citations on every claim.

6. 6

   Evaluate and close the loop

   Build a golden set of 100+ real questions with known answers, track retrieval hit rate and answer groundedness weekly, and route thumbs-down answers into a triage queue that fixes the corpus, not just the prompt.

Common pitfalls

The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.

Frequently asked questions