How to Audit an AI Vendor Before and After You Buy

Vendor audits for AI tend to focus on capability, which is the easiest thing to evaluate and the least likely to cause problems later. This playbook covers what actually matters, drawing on FISTA Solutions' AI enablement work. This article is general guidance, not legal advice.

When is this worth doing?

Before signing, at renewal, and when a vendor's service materially changes — a new model, a new sub-processor, or an acquisition.

It is also worth doing retrospectively for vendors already in use who were never assessed, which in most organisations is several.

What does the sequence look like?

Step	Purpose
1. Establish data handling	Where, how long, used for what
2. Ask about model change	Notice, versioning, deprecation
3. Require evidence	Certifications, tests, incident history
4. Map sub-processors	Your supply chain extends through them
5. Agree and test exit	Export, deletion, transition
6. Review annually	Vendors change quietly

Step 1 — Establish data handling first

Where is data processed, what is retained and for how long, is it used to train models, who can access it, and what happens on deletion.

These answers determine your own compliance position, and they are frequently configurable. Default terms and enterprise terms differ substantially, and the better option is usually available on request rather than by default.

Get the answers in writing in the contract rather than from a sales conversation. Documentation pages change without notice; contracts do not.

Step 2 — Ask how models change and what notice you get

Does the vendor pin model versions, how are updates communicated, what notice is given before deprecation, and can you stay on a version.

This is the question most buyers do not ask and most regret. A vendor that updates the model behind a service without notice has an uncontrolled change process running inside your system.

A notice period is negotiable and worth real leverage, particularly for regulated or validated systems where an unannounced behaviour change is a control failure. See how to handle ai model deprecation.

Step 3 — Require evidence rather than assurances

Security certifications with their scope stated, penetration test summaries, incident history and notification commitments, and evidence behind capability claims.

Scope matters more than the certificate. A certification covering a subsidiary or a different product tells you little about the service you are buying, and vendors are rarely volunteering that distinction.

For capability claims, ask how they were measured and on what data. Claims benchmarked on public datasets say little about your documents. See how to run an ai vendor bake-off.

Step 4 — Map the sub-processors

Which model providers, hosting providers, and other services the vendor relies on, and how you are notified when that list changes.

Your supply chain extends through them. A vendor passing your data to a model provider means your obligations follow it there, and a vendor who cannot tell you who their providers are is not one to rely on for sensitive work.

Require notification of changes. Sub-processor lists change quietly and the change can move your data somewhere you had excluded.

Step 5 — Agree exit terms and test them

Data export in a usable format, confirmed deletion, a transition period, and continued service during it.

Then test the export while the relationship is healthy. An export format discovered to be unusable during a transition is a problem with no good solutions, and vendors are considerably more helpful before a termination notice than after.

For regulated contexts, exit provisions are frequently a formal requirement rather than good practice. See AI and DORA regulation.

Step 6 — Review annually and on change

Re-run the assessment at renewal and whenever the vendor announces something material: a new model, an acquisition, a change of hosting, or a sub-processor change.

Vendors change quietly. An assessment done at signing describes the vendor as they were, and three years is long enough for most of it to be wrong.

Keep the assessment record so each review starts from the previous one rather than from scratch.

What if the vendor will not answer?

That is an answer. A vendor unwilling to state where data is processed, who their sub-processors are, or what notice they give on model changes has told you how the relationship will run.

For low-stakes uses that may be tolerable. For anything involving customer data or regulated decisions, it is a reason to look elsewhere, and the willingness to walk away is what makes the questions effective.

How much diligence is proportionate?

Scale it to what the vendor touches. A tool processing customer data in a regulated workflow warrants full assessment; a productivity tool used on public material does not.

Having tiers written down keeps the process usable. Organisations that apply full diligence to everything either slow adoption to a halt or, more commonly, watch teams buy things on expense claims instead. See how to consolidate ai vendors.

Who needs to be involved?

Someone with security standing, someone with commercial standing, and the technical owner who knows what the service will touch.

Assessments run without the technical owner ask generic questions and miss the specific ones that matter for the actual use.

How long does it take?

Two to four weeks for a full assessment including evidence review and contract negotiation. Shorter for lower tiers, and the tiering is what keeps it manageable.

What are the common failure modes?

Focusing on capability. Accepting assurances without evidence. Not asking about model change. Ignoring sub-processors. Untested exit terms. And assessing once at signing.

How do you know it worked?

Data handling documented in the contract, model change notice agreed, an exit you have tested, and an assessment record you can update rather than rebuild.

What does it cost?

Mostly people's time rather than tooling. The expensive version is the one that stalls halfway and leaves the organisation with neither the old state nor the new one, which is why a narrow first pass beats a comprehensive plan nobody finishes.

Budget the work as an operated change rather than a project with an end date, because most of these need a maintenance tail. See AI total cost of ownership.

What should you do first?

Ask your most important AI vendor where your data is processed and what notice you get before a model changes. The quality of the answer tells you most of what you need.

How FISTA Solutions helps

FISTA Solutions runs this work alongside client teams rather than around them: vendor assessment focused on data handling, model change notice, and exit rather than on capability claims, with evidence required rather than assurances, evidence produced as the work proceeds, and handover that leaves your people able to continue without us. Delivery runs through AI agents, AI enablement, and forward deployed engineers. The record is 150+ projects for 50+ companies across 12+ countries, with 47% average efficiency gains where measured.

To run this with support, message FISTA on WhatsApp, or read how to review an AI contract.