AI Agent Production Readiness Checklist

This checklist is the condensed, operational form of the LLM production readiness whitepaper and the agentic AI governance whitepaper, applied to agents that take actions. Use it as a build plan from the first sprint and as the agenda for the readiness review before launch. Every item should be answered with evidence: a document, a test result, a dashboard, or a named person. FISTA Solutions applies it to every agent delivered through its AI agents practice.

Who should use this checklist?

Engineering owners preparing an agent for launch, business owners who will be accountable for it, and reviewers from risk, security, or governance who must sign off. It applies to agents built in-house and to agents delivered by partners.

Is the specification complete and signed?

1. Purpose and scope are written, including what the agent does not do.
2. Inputs, outputs, and business rules are specified with schemas.
3. Prohibited actions are listed explicitly.
4. Confidence and escalation rules define where uncertain cases go.
5. Approval gates are assigned by consequence class.
6. Quality thresholds and acceptance criteria are defined with measurement method.
7. The business owner and engineering owner have signed the specification.

Method: how to write an ai spec.

Are permissions and tools bounded?

1. The agent has its own identity and short-lived credentials.
2. Every tool has a contract, schema, and consequence level.
3. Permissions are least privilege and match the specification, with read and write separated.
4. Where the agent acts for users, authorization is delegated and user-scoped.
5. No shared service accounts; credentials are revocable independently.
6. Execution of code, files, and network calls is sandboxed with egress controls.

Reference: how to build tool use for llm agents and the AI agent security architecture whitepaper.

Are gates and oversight real?

1. Consequential actions route to approval gates enforced in code.
2. The review queue is staffed, with SLAs and context for reviewers.
3. Reviewers have authority to reject and to trigger autonomy reductions.
4. The autonomy level at launch is recorded, with evidence criteria for the next level.
5. Gate metrics (approval speed, agreement rates) are monitored for ceremonial review.

Reference: how to build a human review queue.

Does evaluation pass?

Check	Evidence
Golden dataset covers every spec category and known failures	Dataset version and coverage report
Quality thresholds met per category	Latest evaluation report
Trajectory evaluation covers actions, arguments, gates, efficiency	Harness results
Safety suite passes: injection, leakage, prohibited actions	Safety report
Graders calibrated against human labels	Calibration record
Suite runs in CI and blocks release	Pipeline configuration
Non-determinism handled with pass rates and pinned versions	Harness configuration

Reference: how to build an agent evaluation harness and the AI evaluation and testing whitepaper.

Is the agent reliable under failure?

1. Budgets on steps, tool calls, tokens, cost, and time are enforced, with defined behavior on breach.
2. Loop detection is in place.
3. Fallbacks for provider outages, timeouts, and malformed outputs are implemented and tested.
4. Idempotency protects write actions against duplicate effects.
5. Load testing at projected peak has been run with realistic tasks.
6. Model and prompt versions are pinned; provider updates trigger re-evaluation.

Reference: how to build an llm gateway.

Is the agent observable and auditable?

1. Trajectory tracing captures inputs, context, model calls, tool calls, validation, gates, outputs, and cost.
2. Operational metrics and quality sampling are live with dashboards.
3. Alerts route to named owners with runbooks.
4. The audit trail is append-only, redacted, retained per policy, and reconstructable.
5. Sensitive data in traces is redacted and access is controlled.

Reference: the AI observability whitepaper and how to build an ai audit trail.

Are data handling and privacy settled?

1. Data classes the agent touches are classified with handling rules.
2. Retrieval enforces permissions; the agent never sees unauthorized content.
3. Provider terms are consistent with the data; redaction or private deployment where required.
4. Retention and residency rules are implemented.
5. Privacy review is complete where personal data is involved.

Reference: ai data privacy compliance.

Is the operating model in place?

1. A named engineering owner and named business owner exist and are on the escalation path.
2. Runbooks cover provider outage, quality regression, cost spike, and safety incident.
3. An incident playbook with containment (autonomy drop, credential revocation) is written.
4. Change control routes prompt, model, tool, and retrieval changes through the evaluation gate.
5. Rollback to the previous approved version is tested.
6. The agent is in the agent register with owner, autonomy level, and review date.

Reference: ai incident response checklist.

Has the launch plan been agreed?

1. Launch mode is shadow or assist unless evidence supports more.
2. Graduation criteria for each autonomy level are written.
3. Baseline metrics for the workflow exist for value measurement.
4. Users and reviewers are trained and know how to escalate and give feedback.
5. A review date for the first post-launch assessment is set.

Reference: ai pilot to production and the AI change management whitepaper.

How should the review be run?

Assemble evidence per section before the meeting; the review confirms it rather than discovering gaps. For each failing item, decide: defer launch, scope down to an autonomy level where the item is not required, or accept with a recorded owner and date. Record the decision in the agent register. Reviews are repeated for material changes.

How FISTA Solutions uses this checklist

FISTA Solutions builds every agent against this checklist from the first sprint, so the readiness review is a confirmation rather than an inspection. The AI agents practice delivers the agent with its evidence package, AI enablement provides the gateway, evaluation, observability, and audit platform the checklist assumes, and forward deployed engineers run the review with your owners and governance function. The record behind the approach is 150+ projects with 99.9% uptime.

To run a readiness review on an agent you are about to launch, message FISTA on WhatsApp, or read why ai agents fail in production for the failures this checklist prevents.