How to Migrate Between LLM Providers Without Breaking Agents

The difference between a migration that takes days and one that takes a quarter is whether the platform was built for it: a gateway that abstracts providers, prompts written to travel, and a regression suite per workload that says whether the new model is at least as good on the cases that matter. This playbook assumes those exist or builds them along the way. It applies the multi-model strategy whitepaper and the governance in model deprecation risk management.

Step 1: What is the migration scope?

From the gateway's logs and the model inventory, list every workload on the outgoing model: applications, routing rules, prompts and versions, tools, retrieval indexes and their embedding models, fine-tuned assets, and logging or evaluation dependencies on vendor tooling. Assign an owner per workload. Decide the order: low-risk, high-volume workloads first to learn; high-consequence workloads last, once the team has practiced the sequence on cases where a mistake is cheap.

Step 2: Do regression suites exist?

Each workload needs a golden set with category-level thresholds and a harness that runs candidates with the same prompts, tools, and retrieval. Where a suite does not exist, build it before qualifying anything; migrating without one means quality is unverified. The method is in the evaluation-driven development whitepaper.

Step 3: Qualify the replacement

Run the suite against the replacement model. Compare quality by category, latency at the percentiles that matter, and cost per completed task. Check the enterprise criteria: tool-use reliability, structured-output validation rate, injection resistance, and data terms, per how to choose an LLM for enterprise agents.

Step 4: How should prompts be adapted?

Symptom in the suite	Likely cause	Adaptation
Format failures	Model-specific formatting assumptions	Strengthen schema instructions; rely on validation and retries
Category regressions	Instructions the old model inferred, the new one needs stated	Make implicit rules explicit
Over- or under-verbosity	Style defaults differ	Length and style instructions
Tool-call errors	Parameter conventions differ	Schema descriptions; examples
Refusals or over-caution	Policy differences	Rephrase intent; supply context

Adapt only what the suite shows degraded, re-run, and repeat until every category clears its threshold. Keep adaptations in a per-model layer so the base prompt stays portable.

Step 5: Stage in the gateway with shadow traffic

Add the replacement as a shadow route: production requests are sent to both models, the new model's outputs are logged and compared but not returned. Shadow traffic reveals inputs the golden set never covered. Review differences by category and add verified cases to the suite. The gateway mechanics are in the LLM gateway architecture whitepaper.

Step 6: Handle embeddings and other assets

Re-embed affected indexes with the new embedding model, store versions, validate against the retrieval golden set, and keep both indexes live during the shift. Rebuild any fine-tuned asset from kept training data if it is worth keeping; otherwise replace it with prompting and retrieval. Point logging and evaluation pipelines at the gateway, not the vendor.

Step 7: Shift traffic gradually

Route a small share to the new model; watch category-level quality from production sampling, error rates, latency, and cost; increase in steps; keep rollback to the old route instant. Consequential agent actions stay behind their approval gates throughout, which also makes the shift safe.

Step 8: Confirm and retire

When production sampling shows quality at the bar across categories and the shift is complete, remove the old model from routing, update the inventory and the deprecation calendar, adjust contracts, and archive the old index once retrieval on the new one is confirmed.

How do agent workloads differ from single-call workloads?

Agents chain model calls with tool use and approvals, so a migration changes behavior across a whole trajectory rather than one response.

Concern	What to check during qualification
Tool-call parameters	Accuracy against schemas; behavior when a parameter must be inferred
Planning and step count	Steps and tokens per task; a model that reaches the same outcome in twice the steps doubles cost
Error handling	Recovery when tools fail or return unexpected content; escalation when appropriate
Approval interactions	The agent still pauses at gates and presents the same evidence
Injection resistance	Adversarial scenarios in the suite produce refusals or escalations, not actions
End-state correctness	Sandboxed runs verify the resulting records, not just transcripts

Qualify agent workloads with end-state verification in a sandboxed copy of their systems, as described in how to build an agent evaluation harness, and hold consequential permissions at their existing gates throughout the shift.

What does the communication plan look like?

Tell workload owners the schedule and their responsibilities before qualification starts; tell approvers and exception handlers when traffic shifts, because output style may change and their queues may look different for a week; tell governance the exposure before and after. Keep a single status page with each workload's stage, so nobody asks whether their agent has moved. Silent migrations generate the complaints that give migration a bad name, even when quality held.

Step 9: What should be recorded?

The prompt adaptations per workload, the regressions found and how they were fixed, the shadow-traffic differences, and the timeline. The next migration starts from this record and is faster for it.

What are the common mistakes?

1. Migrating without regression suites.
2. Rewriting prompts wholesale instead of adapting by evidence.
3. Big-bang cutover.
4. Embeddings forgotten until retrieval degrades.
5. Accepting a category regression to hit a date.
6. Logging left on vendor tooling, so visibility disappears mid-migration.
7. No owner per workload, so qualification stalls on the platform team, which cannot judge what correct means for a workload it does not run.

How does FISTA Solutions help?

FISTA Solutions is an official Anthropic partner and executes provider migrations through forward deployed engineers working inside client teams, on the gateway and evaluation platform its AI enablement practice establishes, so every AI agent moves with quality measured at each step. FISTA has delivered 150+ projects for 50+ companies across 12+ countries with 99.9% uptime.

To plan a migration, message FISTA on WhatsApp, or read LLM vendor lock-in to assess how hard yours will be.