AI Disaster Recovery Checklist: Planning for Provider Failure

AI systems fail in ways conventional disaster recovery plans do not anticipate. This checklist adds the missing scenarios, drawn from FISTA Solutions' AI enablement operational work.

What scenarios need a plan?

Six, with the manual path underpinning all of them.

Scenario	Required response
Provider outage	Failover or graceful degradation
Sustained rate limiting	Queue, shed load, or degrade
Model withdrawn	Migration under deadline
Quality collapse	Detection, then rollback
Corpus or index loss	Restore from backup
Total AI unavailability	Manual path, still working

Provider outage and degradation

The most likely scenario and the one most worth automating. See what is a fallback chain.

• Behaviour on provider timeout and error is defined
• A secondary provider or model is configured and tested
• Failover is automatic where appropriate, manual where not
• Degraded mode defined: what still works without the model
• Users are told when the system is degraded
• Queueing for non-urgent work rather than failing outright
• Provider status monitoring feeds your alerting

Quality collapse

The failure that produces no error. See how to monitor AI quality in production.

• Quality metrics monitored continuously, not only at release
• Alert thresholds defined on quality, not only on errors
• Production sampling reviewed by a person on a schedule
• Rollback to a previous model or prompt version is one action
• Rollback has been tested, not only documented
• A decision threshold agreed for disabling the feature
• User-reported quality problems reach the team quickly

Model deprecation

A deadline set by someone else. See how to run a model migration.

• Deprecation notices subscribed to and monitored
• Current model versions inventoried across all systems
• Evaluation suite ready to run against a replacement
• Migration effort estimated before it is needed
• Capacity reserved for periodic migration work
• An abstraction layer makes switching a configuration change
• A worked example of a past migration documented

Data and index recovery

Several of these exist only in a vendor's system by default.

• Source corpus backed up independently of the index
• Index rebuild from source tested and timed
• Evaluation cases and results backed up
• Prompts and configuration in version control
• Embeddings recoverable or re-computable within an acceptable window
• Vendor-held data exported on a schedule
• Restore tested, not just backup

The manual path

The fallback underneath everything else.

• The manual procedure is documented and current
• Enough people still know how to perform it
• Access to the systems needed for manual work is retained
• Capacity to handle the volume manually is understood
• A threshold defined for switching to manual
• The switch has been rehearsed
• Customer communication for extended degradation is prepared

Plan maintenance

An untested plan is a hypothesis.

• Each scenario has a named owner
• Runbooks exist and are reachable during an outage
• Contact paths for the provider are known and current
• Recovery objectives defined per system and agreed with the business
• A rehearsal is scheduled at least annually
• Lessons from real incidents fold back into the plan
• The plan is reviewed when the architecture changes

What are the most common failures?

Planning for infrastructure failure and not provider failure. No quality alerting. Untested rollback. A manual path nobody can perform any more. And backups of the index but not of the source corpus.

Who should own this?

The system's business owner agrees the recovery objectives; engineering owns the technical fallbacks; operations owns the runbooks and the rehearsal. The manual path is owned by the function that used to do the work.

How often should it run?

Reviewed quarterly, rehearsed at least annually, and updated whenever the architecture or the provider changes. Rollback should be exercised more often than that — ideally during ordinary operations.

What evidence should it produce?

Rehearsal records, rollback test results, restore timings, and the current runbooks. That set demonstrates the plan is real rather than aspirational.

How much of this applies to a low-risk system?

Scale it to consequence. An internal tool that saves time needs a documented degraded mode and nothing more; a system customers depend on needs the full set.

The item that applies at every level is the manual path, because the question of what happens if this stops is asked about every system eventually. See how to staff an AI support rotation.

What should you do first?

Ask whether anyone has tested rolling back your production AI system to a previous version. If not, that is an afternoon's work and the most useful item here.

How FISTA Solutions helps

FISTA Solutions builds and operates production AI systems through AI agents, AI enablement, and forward deployed engineering: quality alerting alongside availability alerting, rollback exercised during ordinary operations, and a manual path kept current and rehearsed, decisions documented with their reasoning, and handover that leaves your team able to maintain what was delivered. The record is 150+ projects for 50+ companies across 12+ countries.

To adapt this checklist to your environment, message FISTA on WhatsApp, or read what is a fallback chain.