AI Vendor Evaluation Checklist

This checklist is the operational form of the AI vendor due diligence whitepaper. It gives buyers the questions to ask and the evidence to demand, organized so that method, security, and terms act as gates before commercial comparison begins. FISTA Solutions publishes it because it expects to be evaluated this way, and because buyers who apply it choose better partners. Related guidance is in how to evaluate ai vendors and questions to ask ai development company.

Who should use this checklist?

Procurement, technology, and business leaders evaluating AI development partners or AI products, and the security, legal, and finance functions supporting them.

Does the vendor have capability evidence?

1. Comparable production work is described with problem, approach, what shipped, measured results, and what went wrong.
2. The specific engineers or product team for your engagement are named, with production history.
3. You can hold technical conversations with those people, not only sales engineers.
4. For products, evaluation results on tasks like yours are available and you can test on your data.
5. Client claims are verifiable; unverifiable claims are treated as absent.

Reference: what to look for in an ai partner.

Does the vendor have a method? (Gate)

Question	Evidence required
How do you define correctness for a system like ours?	Specification from comparable work with acceptance criteria
How do you measure it?	Evaluation report with metrics by category and threshold rationale
How do you build the evaluation dataset?	Description of domain-expert labeling and coverage
How do you launch systems that act?	Shadow and autonomy graduation practice
How do you handle model and provider changes?	Re-evaluation and change-control process
How do you test safety?	Adversarial suite description and results

Reference: the spec-driven development for AI whitepaper and the AI evaluation and testing whitepaper.

Are security and data handling sound? (Gate)

1. Data flows are diagrammed, including model providers and subprocessors, with terms.
2. Access model: SSO, MFA, least privilege, time-bound grants, device policy.
3. Asset ownership: code, infrastructure, and data in your accounts from day one.
4. Secrets and logging practices with redaction.
5. Model-provider terms: no training on your data; retention; residency.
6. AI-specific defenses: prompt injection and leakage testing approach.
7. Audit reports or equivalent evidence where available; incident history and process.

Reference: soc2 for ai vendors and ai vendor security questionnaire.

Are IP and commercial terms acceptable? (Gate)

1. IP assignment of all custom work; vendor pre-existing IP identified and licensed clearly.
2. Data use restrictions on vendor and providers, with retention and deletion.
3. Exit and transition assistance; export of data, prompts, configurations, evaluation sets.
4. Change notification for models, subprocessors, and terms.
5. Pricing explained by drivers; no outcome or timeline guarantees before scoping.
6. Liability and indemnity proportionate to the engagement.
7. Governing law in your jurisdiction where feasible.

Reference: how to negotiate an ai development contract and the outsourcing contract checklist.

Can the vendor operate what it builds?

1. Monitoring approach covers quality, cost, and drift, not only uptime.
2. Support model with response times and escalation.
3. Incident process for AI-specific incidents.
4. Change management for prompt, model, and retrieval changes through evaluation gates.
5. Handoff practice: documentation, runbooks, training, transition period.

Reference: the AI observability whitepaper and the ai project handoff checklist.

Is the vendor stable and verifiable?

1. Legal identity, jurisdiction, founding date, size, and leadership tenure are confirmed.
2. Team depth in the skills you need is confirmed.
3. Continuity practice when an engineer changes is described.
4. For cross-border vendors, the contracting entity and governing law are clear.

Reference: the cross-border engineering delivery model whitepaper.

Do references confirm outcomes?

1. References from comparable projects are provided, including ones with difficulties.
2. Reference calls cover measured results and how they were measured.
3. Problem handling: what went wrong and how the vendor responded.
4. Team continuity: did the engineers who started also finish?
5. Independence after handoff: can the client operate the system alone?

Are there red flags?

1. Demos without production references.
2. No clear answer on defining and measuring correctness.
3. Guarantees before scoping.
4. Reluctance to name engineers.
5. Vendor-owned repositories or infrastructure.
6. Vague data-use terms.
7. Pressure to skip discovery or specification.

Reference: red flags outsourcing ai.

Should you run a pilot?

For significant purchases, structure a paid, scoped pilot with a defined problem, acceptance criteria, evaluation methodology, IP assignment, buyer-owned assets, fixed duration, and exit criteria. The deliverable includes a specification and an evaluation report. Reference: how to structure an ai pilot agreement.

How should the evaluation be scored?

Score each section on evidence quality. Method, security and data, and IP and terms are gates with minimum thresholds. Weight the remaining sections by your risk profile, and compare total cost of ownership over three years rather than quotes. Document the scoring so the decision can be explained.

How FISTA Solutions measures against this checklist

FISTA Solutions is a US-registered company in Delaware with engineering in Faisalabad, names its engineers on every engagement, provides specification and evaluation artifacts from comparable work, keeps code and infrastructure in client accounts, assigns IP and restricts data use in writing, and offers references on measured outcomes. Its record is 150+ projects for 50+ companies across 12+ countries with 99.9% uptime and 47% average efficiency gains, and it does not claim what it cannot verify. Engagements run through forward deployed engineers, AI enablement, AI agents, and staff augmentation.

To put FISTA through this checklist, message us on WhatsApp, or read choose ai development company for the broader selection guide.