Red Flags in an AI Vendor Pitch

AI vendor pitches follow recognizable patterns, and so do the problems they conceal. This guide gives twelve red flags, what each usually indicates, and the question that surfaces it while you are still in the room.

The twelve red flags

Red flag	What it usually conceals	Question that exposes it
A single accuracy number	No real evaluation practice	"On what set, how many cases, labeled by whom, when?"
Reluctance to be tested on your cases	The demo is the product	"Can you run this on fifty of our real cases next week?"
No named production customers doing what you need	You would be the first	"Who is doing exactly this, at what volume, for how long?"
Vague data-use answers	Terms you will not like	"What does the contract say about training, retention, and regions?"
No answer on model changes	Thin product; disruption passed to you	"What happened the last time a provider deprecated a model?"
Demo only on clean inputs	Fails on real messiness	"Show me it on a bad scan and an ambiguous case"
No permission model	Security review will block it	"What can it read and write, and how is that scoped per user?"
Claims of learning from your usage	Either untrue or a data-use issue	"What exactly is stored, where, and can we delete it?"
No mention of human review or exceptions	Exception burden lands on you	"What share of cases will my team still handle, and how do they arrive?"
Pricing that hides usage	Cost surprise at scale	"What is cost per task at our volume, and at three times it?"
No exit path	Deliberate lock-in	"Walk me through a migration away from you"
Pressure to buy before evaluating	Confidence in the pitch, not the product	"We will decide after the evaluation; is that acceptable?"

Why is evaluation refusal the strongest signal?

Because every other issue can be investigated afterward, and this one cannot be worked around. A vendor whose system performs well on real cases has an incentive to be tested: it differentiates them from competitors selling demos. A vendor who deflects ("our benchmarks show," "that would require a paid pilot," "our other customers are satisfied") is protecting a gap.

The request is reasonable and should be standard: fifty to a hundred of your real cases with known correct outcomes, run by them or by you, with results reported by pass rate and failure type. The how executives should evaluate an AI demo guide covers converting a pitch into an evaluation.

Why do accuracy claims dissolve under questioning?

Because most are produced on curated sets by the vendor's own team with no defined threshold. Ask the five questions in the table and the number usually becomes "we don't have that broken down." That is not necessarily disqualifying, but it means the claim was never evidence. The how to read an AI evaluation report guide covers what real evidence looks like.

Why does the model-change question matter?

Because it reveals how much product exists around the model. A vendor with real engineering has been through a deprecation, has evaluation sets, revalidated, and notified customers. A thin wrapper experiences provider changes as an outage and passes the disruption downstream. Ask what happened the last time, specifically. The model deprecation risk management guide covers what good handling looks like.

What about the exception question?

Most AI products handle a portion of cases and route the rest. The portion is the value, and the routing determines the burden. A vendor who has not addressed exceptions has left the operational work with you and has not counted it in the business case. Ask what share of cases the buyer's team will still handle, in what form they arrive, and what context is included. The document AI explained for executives piece covers why exception design determines whether the system reduces work.

Are any of these disqualifying on their own?

Evaluation refusal and evasion on data use, usually yes. The rest are signals to investigate: a young vendor may have no named customers doing exactly this and still be the right choice with appropriate terms and a pilot structured as a real evaluation. The purpose of the list is to know what you are accepting rather than to eliminate candidates automatically.

What should buyers do with the answers?

Put the important ones in the contract: data use, model change notification, accuracy commitments where offered, exception volumes, exit assistance, and export formats. Statements made in a pitch have no force; the same statements in a contract do. The how to sunset an AI vendor guide covers the exit terms specifically.

What should executives ask?

• Will you run this on our cases, and when?
• What set produced your accuracy figure, and who labeled it?
• Who is doing exactly this in production, and may we speak to them?
• What does the contract say about our data?
• How would we migrate away from you?

How can FISTA Solutions help?

FISTA Solutions runs independent evaluations of vendor systems on clients' own cases, assesses model dependency, exception burden, and exit terms, and builds the alternative where a build is the better answer, through its AI enablement and AI agents practices. Since 2017, FISTA has delivered 150+ projects for 50+ companies across 12+ countries.

To have a vendor's claims tested on your own cases before you sign, talk to FISTA on WhatsApp, or read how to evaluate AI vendors.