Playbook

Vendor bake-offs get won by whoever demos best unless the test is defined first. This playbook covers designing an evaluation on your own data, normalising scope, and scoring against criteria set in advance.

Quality thresholds only matter if something happens when they are breached. This playbook covers anchoring thresholds to the process being replaced, setting them per category, and attaching consequences.

Most AI centres of excellence become review boards that slow delivery. This playbook covers giving one a mandate that produces capability, what it should own, and how to keep it from becoming a gate.

Decommissioning is the step organisations skip, leaving orphaned systems processing data nobody watches. This playbook covers confirming dependencies, handling data and records, and closing arrangements cleanly.

Shadow deployment is the cheapest way to learn how a change behaves on real traffic. This playbook covers mirroring safely, comparing outputs meaningfully, controlling the extra cost, and deciding to promote.

Champion-challenger testing answers whether a change helps real users, which shadow deployment cannot. This playbook covers metric choice, traffic splitting, run length, and deciding without post-hoc reasoning.

Agents that work at pilot volume fail at scale for specific reasons: provider limits, cost per task, and quality variance across a wider input distribution. This playbook covers each.

AI business cases fail scrutiny when the baseline is assumed and the running cost is missing. This playbook covers building a case from a measured baseline, an explicit mechanism, and costs that include operation.

Debugging an agent is harder than debugging a service because the control flow is decided at runtime. This playbook covers reproducing trajectories, isolating the failing step, and the distinct failure classes.

Most RAG quality problems are retrieval problems dressed as generation problems. This playbook covers diagnosing which layer is failing, fixing retrieval properly, and only then working on the answer.

Changing provider involves more than swapping an endpoint: interfaces differ, behaviour differs, and contracts and data handling change too. This playbook covers the sequence that avoids surprises.

AI latency problems are usually not the model. This playbook covers measuring the real distribution, attributing time to retrieval, context, and calls, and the changes that help most without quietly degrading quality.

Production readiness reviews for AI need to cover things conventional checklists miss. This playbook covers what to examine, who should be in the room, and what constitutes a blocking finding.

AI costs drift upward quietly because nobody owns them. This playbook covers measuring cost per task, finding the patterns that dominate spend, routing work by difficulty, and preventing the savings from eroding.

AI incidents are harder to reconstruct than outages because the system did not fail — it was wrong. This playbook covers reconstruction, cause analysis that goes past the prompt, and producing changes that get made.

AI vendor sprawl happens quickly and consolidating it badly drives usage underground. This playbook covers building the real inventory, deciding what survives, and migrating teams without losing capability.

Retrieval systems inherit whatever is wrong with the knowledge base. This playbook covers establishing ownership and currency, restructuring for retrieval, removing contradictions, and the controls that stop decay.

Agents that work in demos fail in production for predictable reasons. This playbook covers scoping authority, building evaluation before launch, staged rollout with real users, and the monitoring that catches problems first.

Data cleanup projects run forever when nobody defines what clean means. This playbook covers scoping to a specific use, measuring what actually blocks it, fixing at source where possible, and stopping deliberately.

Evaluation programmes fail when they measure the wrong thing or nobody acts on the results. This playbook covers building the first evaluation set, agreeing what good means, running it continuously, and closing the loop.

AI red teaming finds different problems from security testing. This playbook covers scoping the exercise, who should participate, the attack categories worth covering, and turning findings into controls rather than a report.

Most prompt libraries are collections nobody opens. This playbook covers what belongs in one, how to version and test entries, who maintains it, and how to tell whether anyone is actually using it.

Model migrations fail when nobody established what the old model did. This playbook covers building a behavioural baseline, shadow running the candidate, comparing outputs honestly, and cutting over reversibly.

Most AI training programmes teach tools and change nothing. This playbook covers segmenting the audience by task, building curriculum around real work, providing sanctioned tooling, and measuring whether behaviour actually changed.