How to Build an Agentic RAG System (Playbook)

Standard RAG retrieves once and generates. That works for direct questions and fails on complex ones: multi-part questions, questions whose right source is not obvious, questions where the first retrieval reveals what to search next. Agentic RAG adds planning, iteration, source selection, and verification, at a cost in latency and spend that must be justified. This playbook covers when it helps and how to build it, following FISTA's AI agents practice. Foundations are in how to build a rag system and the enterprise RAG reference architecture whitepaper.

What does the system do?

Stage	Function
Routing	Decide whether the question takes the standard or agentic path
Planning	Decompose into sub-questions; choose initial sources
Iterative retrieval	Retrieve, assess, reformulate, retrieve again within budget
Source selection	Choose among indexes, databases, APIs, and tools per sub-question
Verification	Check evidence relevance and claim support; surface contradictions
Synthesis	Grounded answer with citations, reasoning summary, and gaps
Budgets	Iteration, call, token, cost, and time limits
Evaluation	Complex question set; trajectory metrics; baseline comparison

Step 1: Establish the baseline and the need

Build or measure a strong hybrid RAG baseline. Classify questions by complexity and measure where the baseline fails. Agentic retrieval is justified for the failing categories only. See how to build a hybrid search system and how to improve rag accuracy.

Step 2: Build the router

Classify incoming questions by complexity and route simple ones to the standard path and complex ones to the agentic path, with confidence and a fallback. The router keeps cost and latency proportional. Concepts are in what is an llm router.

Step 3: Implement planning

For agentic-path questions, produce a plan: sub-questions, dependencies between them, candidate sources per sub-question, and stopping criteria. Constrain the plan with a schema and a maximum size. Planning concepts are in what is planning in ai agents and what is an agentic workflow.

Step 4: Retrieve iteratively with source selection

For each sub-question, choose the source (document index, structured database, API, or tool), retrieve, assess relevance and coverage, and reformulate or switch sources when results are poor. Record every retrieval with provenance. Structured sources answer numeric and factual sub-questions better than passages. Tool design is in how to build tool use for llm agents.

Step 5: Verify

Before synthesis, check that retrieved evidence is relevant and authoritative, that draft claims are supported by cited passages through entailment or quote checks, that contradictions between sources are identified, and that unresolved sub-questions are marked as gaps. Verification is deterministic where possible and uses calibrated model judges otherwise. See what is groundedness in ai and what is llm-as-a-judge.

Step 6: Enforce budgets

Set per-question limits on iterations, retrieval calls, tokens, cost, and wall-clock time; detect repeated queries and states; and define behavior on budget exhaustion: answer with explicit gaps or escalate. Budgets are enforced in code, not requested in prompts. Controls are in ai agent guardrails.

Step 7: Synthesize with citations and gaps

Generate the answer from verified evidence with citations per claim, a brief summary of how the answer was assembled, contradictions noted, and a gaps section for what was not found. Validate citations again on the final output. Synthesis patterns are in how to build an ai research assistant.

Step 8: Evaluate trajectories against the baseline

Build a labeled set of complex questions with expert answers and source expectations. Measure answer correctness, coverage, groundedness, citation validity, gap reporting, and trajectory metrics (iterations, calls, cost, latency), and compare against the baseline on the same questions. Include adversarial cases through retrieved content. Harness design is in how to build an agent evaluation harness.

Step 9: Observe and operate

Trace full trajectories with sources and decisions; monitor router accuracy, budget breaches, cost per question by path, and quality samples; and import failures into the evaluation set. Observability design is in the AI observability whitepaper.

Worked example: an internal policy and data assistant

An operations team asks questions that mix policy text with operational data: whether a proposed vendor arrangement complies with procurement policy given the vendor's spend history and risk rating. The standard RAG baseline answers pure policy questions well and fails on mixed ones. The router sends mixed questions to the agentic path. Planning decomposes into policy retrieval, spend lookup from the procurement database, and risk-rating lookup from the vendor system. Iterative retrieval reformulates the policy query after the first pass returns general guidance, finding the specific threshold clause. Verification confirms each claim maps to a passage or a database result and flags that the risk rating is a year old. Synthesis answers with citations to the clause and the records, notes the stale rating as a gap, and stays within its budget. Evaluation shows the agentic path answering the mixed questions the baseline missed at a higher but bounded cost, and the router keeps most questions on the cheap path.

What does it cost to run?

Agentic questions cost several times a standard question in retrieval and model calls; the router and budgets keep total cost proportional to the share of questions that need it. Drivers are in cost of running llms in production.

What are the common mistakes?

• Routing every question through the agentic path.
• Planning without a schema or size limit.
• No budgets, so hard questions run away.
• Skipping verification and synthesizing from irrelevant evidence.
• Evaluating answers only, never trajectories or cost.
• Building agentic retrieval before the baseline is strong.

How FISTA Solutions builds agentic RAG

FISTA Solutions builds agentic RAG systems to this playbook: a strong baseline first, a router that keeps cost proportional, schema-constrained planning, iterative retrieval with source selection and provenance, deterministic and judged verification, enforced budgets, cited synthesis with gaps, and trajectory evaluation against the baseline. The AI agents practice delivers the system, AI enablement the retrieval and gateway platform, and forward deployed engineers embed with your teams to define the question set. The record behind the work is 150+ projects with 99.9% uptime.

To assess whether agentic RAG fits your questions, message FISTA on WhatsApp, or read how to build a graph rag system for the relationship-heavy alternative.