How to Upgrade Basic RAG to Agentic RAG: A Playbook

Basic retrieval-augmented generation has a simple loop: embed the question, fetch the top chunks, generate. It works until the question needs two sources, or the wording does not match the documents, or nothing relevant exists and the model answers anyway. Agentic RAG replaces the single pass with planning, iteration, and verification. This playbook upgrades an existing pipeline in measured stages rather than rebuilding it. It applies the architecture in the enterprise RAG reference architecture whitepaper and the build in how to build an agentic RAG system, and it starts from the failure analysis in why RAG systems hallucinate.

Step 1: Where does the basic pipeline fail?

Before adding agency, diagnose. Sample failed answers and classify them.

Failure	Symptom	Likely fix
Retrieval miss	Right document exists, not retrieved	Indexing, chunking, hybrid search, reranking before any agency
Query mismatch	Question wording differs from document wording	Query rewriting and decomposition (agentic stage 1)
Multi-source need	Answer needs several documents	Iterative retrieval (stage 3)
Missing evidence	Nothing relevant exists; model answers anyway	Groundedness gate and refusal (stage 4)
Wrong source type	Structured data needed, text retrieved	Retrieval tool routing (stage 2)
Stale content	Old version retrieved	Versioning and freshness filters

If most failures are retrieval misses, fix the index first; agency on top of bad retrieval iterates over the same bad results. Record the failure distribution, because it is the baseline every later stage is judged against, and it tells you which stage to build first rather than following a generic order.

Step 2: Build the golden set and score separately

Real questions with verified answers and the sources that support them, by category, including questions with no answer in the corpus. Score retrieval, groundedness, correctness, refusal correctness, steps, cost, and latency independently, so each stage's effect is visible. The method is in how to build a golden dataset.

Step 3: Stage 1, query understanding

Add a planning step: interpret the question, resolve references, decompose multi-part questions into sub-questions, and rewrite each for retrieval. Gate on retrieval quality improvement for the query-mismatch and multi-part categories. Patterns are in how to improve RAG accuracy.

Step 4: Stage 2, retrieval tools and routing

Expose retrieval as tools: semantic search, lexical search, structured lookups, and any specialized indexes, through the governed tool layer. Let the planner choose per sub-question. Gate on the wrong-source-type category. Hybrid retrieval design is in how to build a hybrid search system.

Step 5: Stage 3, iterative retrieval with sufficiency checks

After each retrieval, the agent judges whether the evidence answers the sub-question; if not, it reformulates and retrieves again, within a step budget. Gate on the multi-source category and on cost per answered question; iteration that does not improve correctness is waste.

Step 6: Stage 4, groundedness verification and refusal

Before answering, verify each claim in the draft against retrieved evidence; drop or flag unsupported claims; if the core answer is unsupported, decline or escalate with what was found. Gate on the missing-evidence category and on the rate of confident wrong answers. This stage most improves trust. Verification patterns are in how to ground an LLM.

Step 6b: What about answers that need structured data?

Many enterprise questions mix text and numbers: a policy clause plus a customer's actual balance, a procedure plus the current status of a ticket. Basic RAG cannot answer these because the numbers live in databases, not documents. In the agentic pipeline, structured lookups are simply another retrieval tool: the planner recognizes that a sub-question needs a record or a metric, calls a governed data tool under the asking user's permissions, and combines the result with the retrieved text in the grounded answer. The data tool follows the same discipline as any agent tool, scoped permissions, validated parameters, and logging, described in how to build an MCP server for Postgres, and the governance of data access in the data agents and text-to-SQL whitepaper.

Two cautions apply. Structured answers must state the data's freshness, because a correct query over stale data is a wrong answer for the person asking. And the groundedness gate must treat the structured result as evidence like any other: a claim about a balance is supported only if the tool returned that balance, never because the model remembered a plausible number.

Step 7: How should fast and agentic paths coexist?

Keep basic RAG as the fast path for simple, single-source questions, and route to the agentic path on question complexity signals or a failed sufficiency check. This keeps latency and cost low for the majority and reserves iteration for questions that need it. Routing rules live in the gateway.

Step 8: Roll out and measure

Ship stages in order, each behind the golden-set gate, with production sampling scoring the same measures. Watch steps and cost per answered question; set budgets. Feed production misses into the golden set. Report by category so a stage that helps multi-part questions but slows simple ones is visible.

Step 9: Operate

Re-index on content changes, expire stale versions, re-run the golden set on model or prompt changes, and review refusal rates: too high means retrieval gaps; too low with rising misses means the groundedness gate weakened.

What are the common mistakes?

1. Agency on bad retrieval.
2. Upgrading everything instead of routing by question type.
3. No separate retrieval and answer scoring, so stages cannot be judged.
4. Unbudgeted iteration.
5. Skipping the refusal path, the stage that matters most.
6. Stale content retrieved confidently.
7. Latency ignored, so the agentic path makes every question slow instead of only the hard ones.

How does FISTA Solutions help?

FISTA Solutions upgrades RAG pipelines stage by stage, gated on evaluation, through forward deployed engineers working with your data and knowledge teams, on the retrieval and gateway platform its AI enablement practice establishes, so every AI agent that depends on retrieval inherits the improvements. FISTA has delivered 150+ projects for 50+ companies across 12+ countries.

To diagnose your pipeline's failures, message FISTA on WhatsApp, or read how to build an agentic RAG system for a from-scratch build.