RAG System Launch Checklist

This checklist is the launch-time verification of the enterprise RAG reference architecture whitepaper. It exists because most RAG systems that fail in production were never checked against it: content was indexed without curation, permissions were assumed, retrieval was never measured, and freshness was hoped for. Use it as the agenda for the go-live review, with evidence for every item. FISTA Solutions applies it in its AI enablement practice; the step-by-step build is in how to build a rag system.

Who should use this checklist?

Engineering owners of RAG systems, knowledge and content owners whose material is indexed, security reviewers responsible for access control, and business owners accountable for the assistant's answers.

Is the content curated and owned?

1. Sources in scope are inventoried with owners; excluded sources are documented.
2. Contradictions and stale documents have been resolved or retired.
3. Each source has an authoritative designation per topic.
4. Metadata (type, date, owner, audience, effective date) is present and populated.
5. An update process with owners exists for indexed content.

Reference: the data readiness for generative AI whitepaper.

Is ingestion preserving meaning?

1. Parsers preserve structure: headings, tables, lists, cross-references.
2. Chunking is structure-aware with tested sizes and overlap.
3. Chunks are enriched with title, section path, and metadata.
4. Incremental updates and deletions are captured from each source.
5. Parsing quality has been sampled and reviewed per document type.

Reference: what is chunking in rag.

Are permissions enforced?

1. Access-control information is captured per chunk at ingestion.
2. User entitlements are resolved at query time from the identity provider.
3. Permissions are applied as hard filters before ranking.
4. Compliance tests confirm restricted content never surfaces for unauthorized users.
5. Entitlement re-sync runs on schedule and on change events.

Reference: ai access control.

Does retrieval meet targets?

Check	Evidence
Labeled question set covers question types and topics	Dataset version
Hybrid retrieval (lexical plus vector) is in place	Architecture
Reranking is applied to fused candidates	Configuration
Query rewriting handles context and abbreviations	Test results
Recall and precision at k meet targets by question type	Retrieval evaluation report
Embedding model is evaluated and pinned	Selection record

Reference: how to build a hybrid search system and how to improve rag accuracy.

Is generation grounded?

1. Only reranked, authorized chunks enter the prompt within a token budget.
2. Citations are required and validated to resolve to retrieved passages.
3. Refusal triggers on insufficient evidence with a defined message and escalation.
4. Groundedness is checked on outputs, sampled in production.
5. Output validation blocks commitments, prohibited content, and leakage.

Reference: how to ground an llm and llm output validation.

Does evaluation pass in CI?

1. Retrieval metrics and generation metrics are measured separately.
2. Generation is scored for correctness, groundedness, citation validity, and refusal correctness.
3. Safety tests cover injection through documents and queries.
4. Graders are calibrated against human labels.
5. The suite runs in CI and blocks release on threshold breaches.
6. A held-out set exists; prompts were not tuned on the test set.

Reference: the AI evaluation and testing whitepaper.

Are freshness, cost, and latency under control?

1. Index age per source is visible; freshness lag meets targets.
2. Deletion propagation is tested: removed documents disappear from results.
3. Latency budget per stage is met at projected load, with caching and cutoffs.
4. Cost per query is modeled and tracked, with reranker cutoffs and routing.
5. Provider model versions are pinned with re-evaluation on updates.

Reference: how to build an llm gateway and enterprise rag cost.

Is the system operable?

1. Traces capture query, rewritten query, retrieved and reranked chunks, prompt, answer, citations, latency, and cost, with redaction.
2. Dashboards and alerts exist for quality, latency, cost, freshness, and errors.
3. Feedback capture and unanswered-question logging feed content owners.
4. Named owners and runbooks exist; rollback is tested.
5. Privacy handling for questions containing personal data is settled.

Reference: the AI observability whitepaper.

Is the launch staged?

1. A pilot audience is defined with feedback mechanisms.
2. Success measures and a review date are set.
3. Content owners are ready to act on unanswered-question reports.
4. Expansion criteria are written.

Reference: ai pilot to production.

How should failing items be handled?

Retrieval and permission failures block launch. Grounding and evaluation gaps block launch beyond a closed pilot. Freshness, cost, and operability gaps may be accepted for a pilot with owners and dates recorded. Record decisions and revisit at the post-launch review.

How FISTA Solutions uses this checklist

FISTA Solutions builds RAG systems against this checklist from the start, so go-live is a confirmation of evidence produced during delivery. The AI enablement practice delivers the retrieval platform with permission enforcement, hybrid retrieval, evaluation, and observability built in; AI agents build on it; and forward deployed engineers run curation and evaluation with your content owners. The record behind the approach is 150+ projects with 99.9% uptime.

To run a launch review on a RAG system, message FISTA on WhatsApp, or read why rag systems hallucinate for the failures this checklist prevents.