AI Dependency Review Checklist: Knowing What You Run

AI systems depend on components that change without any release on your side. This checklist covers knowing what you run, drawn from FISTA Solutions' AI agents engineering work.

What are the dependencies?

Six categories, most of them untracked.

Dependency	How it changes under you
Model version	Provider updates or deprecates
Embedding model	Update invalidates the index
Tool server	Its own release cycle
Prompt template	Edited outside review
Orchestration framework	Ordinary package updates
Third-party agent	Entirely outside your control

Inventory

The artefact almost nobody has. See the agent supply chain problem.

• Every external dependency listed per AI system
• Owner named per dependency
• Access scope recorded for anything with credentials
• Version or identifier recorded
• Last review date recorded
• Source and provenance recorded
• Inventory maintained rather than built once

Model and embedding versions

The category that changes without your involvement.

• Model versions pinned where the provider supports it
• Deprecation notices subscribed to and monitored
• Embedding model version recorded with the index
• Re-embedding cost and time estimated in advance
• Evaluation suite ready to run against a new version
• Version logged with every output
• A migration plan exists before it is needed

Tool servers

Review them as privileged code, because that is what they are.

• Each tool server's source and maintainer identified
• Code reviewed where it is not from a trusted internal team
• Credentials it holds enumerated and scoped
• Version pinned rather than tracking latest
• Update process defined with review
• Network access from the server restricted
• Its own dependencies reviewed

Prompts and templates

Code that arrived as configuration. See prompt review checklist.

• Origin of every production prompt known
• Copied templates reviewed line by line
• Prompts in version control with owners
• Tool descriptions reviewed as prompts
• Few-shot examples checked for sensitive content
• Prompt version logged with outputs
• Changes go through review

Frameworks and libraries

Conventional practice, applied to a fast-moving ecosystem.

• Versions pinned in a lock file
• Vulnerability scanning enabled
• Update cadence defined rather than ad hoc
• Breaking changes reviewed before upgrading
• Maintenance status of key libraries assessed
• Transitive dependencies reviewed for anything unexpected
• Licence compatibility confirmed

Change detection

You need to know when something moved.

• Provider change notifications monitored
• Evaluation run on any announced model change
• Behaviour monitoring capable of detecting an unannounced change
• Tool server updates reviewed before deployment
• Alerts on unexpected version changes in production
• Change log covering dependency updates
• Version information available during incident investigation

What are the most common failures?

No inventory. Models unpinned. Tool servers tracking latest. Prompts copied without review. Embedding model changed without re-embedding. And version information absent from logs.

Who should own this?

The team running the system owns its dependency inventory; security reviews anything privileged. An inventory owned centrally without the running team goes stale immediately.

How often should it run?

Reviewed quarterly, and on any provider announcement or tool server update. Full review whenever a new dependency is added.

What evidence should it produce?

The current inventory with owners and review dates, pinned version records, tool server review notes, and version information present in production logs.

What about dependencies inside vendor products?

You usually cannot see them, which is why the vendor assessment asks about their model providers and change notice.

What you can do is record which vendor products form part of your AI systems, and treat their behaviour changes as dependency changes. See AI third party risk checklist.

What should you do first?

Try to list every external dependency of your main AI system with owners and versions. The difficulty of producing that list is the finding.

How FISTA Solutions helps

FISTA Solutions builds and operates production AI systems through AI agents, AI enablement, and forward deployed engineering: a maintained dependency inventory with owners and access scope, versions pinned where possible and logged against every action where not, decisions documented with their reasoning, and handover that leaves your team able to maintain what was delivered. The record is 150+ projects for 50+ companies across 12+ countries.

To adapt this checklist to your environment, message FISTA on WhatsApp, or read the agent supply chain problem.