Agentic RAG Systems
Agentic RAG systems: query planners, retrieval tools, multi-hop RAG loops, iteration limits, a groundedness critic, and when agentic retrieval is wrong to ship.
- Read time
- 16 min
- Level
- Advanced
- Updated
- 2026-08-25
- Components
- Query planner · Retrieval tools · Iteration budget
An agentic RAG system is a retrieval loop in which a planner writes and revises search queries, calls retrieval as a tool, accumulates evidence across hops, and a groundedness critic blocks answers that are not supported — all under a hard iteration and token budget.
Key takeaways
- 01
Agentic RAG is for multi-hop and underspecified questions; single-shot RAG remains the right design for lookup queries with one target document.
- 02
Retrieval must be a tool the model calls, not a hidden pre-step, or the planner cannot reformulate after a miss.
- 03
Cap steps, tokens, and wall-clock on every loop — unbounded agentic retrieval is a cost and latency incident, not a feature.
- 04
A groundedness critic that can force another search or a refusal is what separates agentic retrieval from a chatty search loop.
- 05
Compounding error is the distinctive failure: a bad first hop poisons later queries, so log evidence per hop and evaluate hop-level hit rate.
What agentic RAG adds to single-shot retrieval
Agentic RAG, or agentic retrieval, turns search from a single pre-prompt fetch into a controlled loop. A planner inspects the question, decides what to look up, calls retrieval tools, reads the hits, and either answers, refuses, or searches again. That loop is what multi-hop RAG needs: "what is our refund policy for the SKU on last week's P1 ticket" is two or three lookups, not one embedding query.
It sits on top of a standard RAG architecture — the same ingest, hybrid index, and ACL filter — and should not fork a second corpus. The extra machinery is planning, tool calling, an evidence store, a critic, and budgets. If those are missing, you have retry-the-same-query, not an agentic system.
- Planner: decompose, write queries, decide stop vs. continue
- Tools: keyword, dense, metadata filter, optionally SQL or graph
- Evidence store: hop-level passages with provenance
- Critic plus budget: groundedness check and hard stop
Planner, tools, and the evidence store
The planner should be a narrow role: emit structured next actions (search, filter, answer, refuse), not a free-form essay. Give it retrieval tools with typed arguments — query string, source filter, date range — executed server-side with the same permission path as single-shot RAG. Never let the model see documents the user cannot.
Park retrieved passages in an evidence store outside the scratchpad. Each hop records query, hits, scores, and which spans the planner intends to use. Later hops read that store instead of the full transcript, which keeps the context window from turning into a junk drawer and makes traces replayable.
Multi-hop RAG and iteration limits
Multi-hop RAG earns its keep when the answer depends on an intermediate fact: a ticket ID that yields a SKU that yields a policy clause. The planner must be allowed to issue a different query after reading hop-1 hits; identical query retry is a bug. Cap the loop explicitly — a typical production budget is 3–5 retrieval calls, a token ceiling, and a wall-clock budget aligned to the product SLA.
When the budget expires, the system should return a partial, cited answer or a refusal with what was searched, not a last-gasp ungrounded guess. The orchestrator enforces the cap; the model must not be trusted to stop itself.
The groundedness critic
A groundedness critic is a second check, often a cheaper model with a binary rubric: every claim in the draft is supported by a stored passage, or it is not. Failures route back to the planner ("search for X") or to refusal. Without a critic, agentic RAG optimizes for sounding finished rather than being backed by evidence.
Freeze the critic's prompt and model version, and calibrate it against human labels the same way you would any LLM judge. An unversioned critic that "usually agrees" is a moving gate. Log the verdict and the cited spans; that log is the eval case you will need next month.
Failure modes and when not to use agentic RAG
Distinctive failures are loops (the planner rewrites the same query), query drift (later hops wander off the user's question), critic rubber-stamping, and cost blow-ups on high QPS. Compounding retrieval error is the silent one: hop 1 retrieves a near-miss, hop 2 searches as if that near-miss were fact, and the answer is fluent and wrong.
Do not use agentic RAG for FAQ lookup, latency-critical chat, or high-QPS search where a 200ms retriever already hits. The extra hops buy recall on compositional questions and spend latency and tokens on everything else. Start single-shot; promote a query class to the agentic path when golden-set multi-hop failure rate justifies it.
- Looping and identical-query retries
- Query drift away from the original ask
- Ungrounded answers after budget expiry
- Cost and p95 latency that the product cannot afford
Evaluating agentic retrieval
Evaluate hops and the end answer separately. Hop-level: did retrieval at step n contain a supporting span. End-to-end: faithfulness, citation validity, steps used, cost, and correct refusal. Include a stratum of known multi-hop questions and a stratum of single-hop questions that must not take extra hops — over-searching is a regression. Trace every tool call; without hop traces you cannot tell planner error from index error.