How to Interview an AI Leader

Hiring an AI leader is difficult because the credentials that look impressive on paper correlate weakly with the ability to get systems into production in a real company. Publications, conference talks, and a fluent vocabulary do not predict whether someone can navigate a data access problem, persuade a sceptical business owner, and keep an agent working through a model deprecation. This guide gives the questions that do predict it.

Interview one system, deeply

The most informative interview takes a single system the candidate built and goes as deep as they can go:

Question	What it reveals
What problem did it solve, and for whom?	Whether they think in business terms
What was the baseline, and how was it measured?	Whether they work from evidence
How did you know it worked?	Evaluation practice, or its absence
What did it cost to build and to run?	Whether they own economics
What broke after launch, and how did you find out?	Operating experience, the scarcest signal
What would you do differently?	Self-assessment without defensiveness
Who owns it now?	Whether they build things that survive them

A candidate with real production experience answers these with specifics, including unflattering ones. A candidate whose experience is pilots and presentations produces general answers and redirects to strategy.

Why is operating experience the key signal?

Because building an AI system that demos is common and keeping one working is not. The problems that define the job appear after launch: drift when a provider updates a model, exception rates rising after an upstream change, a business owner losing confidence, an incident at an inconvenient moment. Someone who has lived through those has judgment that cannot be acquired from reading.

Ask specifically: "tell me about a time an AI system you were responsible for degraded in production." A candidate with real experience has a story; one without will generalize about monitoring best practice. The AI agent failure modes for executives piece describes the failures a real answer will reference.

Ask what they decided not to build

Strong AI leaders decline more than they approve: the executive's pet idea with no baseline, the use case whose data does not exist, the multi-agent architecture that a single agent would handle. Ask for two examples and how they handled the resulting conversation.

A candidate who has never declined anything has either never had the authority or never had the judgment, and both are problems for a leadership role.

What are the red flags?

• No baselines for anything they claim to have improved.
• No evaluation practice: quality assessed by demonstration and intuition.
• A career of pilots with nothing named that runs in production today.
• No incidents, ever, which means either no production systems or no detection.
• Vendor-shaped answers: descriptions of what tools do rather than what they built.
• Cannot explain cost: neither build nor run economics.
• All strategy, no delivery: comfortable at the operating-model level, vague below it.

How do you assess without technical depth yourself?

Listen for texture: numbers, dates, names of systems, trade-offs made, things that went wrong. Technical accuracy is not the test; specificity is. A candidate who says "we got the cost per document to a level the CFO accepted by routing the easy ones to a smaller model, after the first month's bill was three times our estimate" has been there. One who says "we leveraged state-of-the-art models to drive efficiency" has not.

Bring a practitioner to the final interview if you can, ideally one from outside the company, and have them probe the technical claims while you assess judgment and communication. The AI leadership roles explained piece covers which responsibility you are actually hiring for, which should shape the weighting.

What should references be asked?

Not whether the candidate was good, which produces nothing. Ask:

• What do the systems they built look like now, a year on?
• Did the business owners actually adopt them?
• How did they handle an incident?
• Were they able to say no to a bad idea from someone senior?
• Did the organization keep the capability after they moved on?

The last question is the most revealing for a leadership hire: leaders who build capability leave something behind, and leaders who build dependence do not.

What should executives ask themselves?

• Which responsibility am I actually hiring: accountable executive, platform lead, or engineering lead?
• Have I asked for one system in depth rather than a career overview?
• Did the candidate volunteer anything that went wrong?
• Can they name what they declined to build?
• What did the references say about systems a year later?

How can FISTA Solutions help?

FISTA Solutions provides forward deployed engineers and applied AI leadership on engagement while companies build internal capability, and its staff augmentation practice supports hiring assessment with practitioners who have taken systems to production. Since 2017, FISTA has delivered 150+ projects for 50+ companies across 12+ countries.

To have a practitioner join your final-round interviews, talk to FISTA on WhatsApp, or read should you hire a chief AI officer.