Checklist ┬╖ 4 minute read
AI Explainability Checklist: Explaining Decisions Honestly
A model's self-explanation is another generation, not an account of how it reached an output. Honest explainability comes from recording the inputs, the retrieved context, the rules applied, and the deterministic logic тАФ and from designing consequential decisions so the explanation is a rule rather than a narrative.
A model's self-explanation is another generation, not an account of its computation. This checklist covers explaining decisions in ways that are actually true, drawn from FISTA Solutions' AI enablement governance work.
What can actually be explained?
Six sources of explanation, in order of reliability.
| Source | Reliability |
|---|---|
| The rule that decided | Exact |
| Inputs received | Exact |
| Retrieved context used | Exact |
| Validation that passed or failed | Exact |
| Attribution analysis | Approximate |
| Model's own narrative | Not a record |
Design for explainability
Decided at architecture time, not added afterwards. See the return of determinism.
- Consequential decisions made by rules, not by model judgement
- The model's role limited to interpretation and language
- Rules expressed in code that can be read and cited
- Decision factors identified and named in advance
- Thresholds documented with their rationale
- Changes to rules versioned and dated
- The explanation path designed before the system is built
Decision records
The factual basis for any later explanation.
- Inputs recorded as received
- Retrieved context recorded with source references
- Model and prompt versions recorded
- Rules and thresholds in force at the time recorded
- Output and action recorded
- Human involvement and any override recorded
- Records retrievable by decision identifier
User-facing explanations
An explanation that enables no action is a disclosure.
- Plain language, no internal terminology
- The main factors stated specifically
- What the person could do differently, where applicable
- How to request human review, clearly signposted
- Consistent with the actual decision logic, not a generated approximation
- Tested with people outside the team for comprehensibility
- Available in the user's language
Internal explanations
Support and reviewers need more detail than end users.
- Support tooling shows the decision record
- Retrieved sources visible and openable
- Rules applied shown with their values
- Reviewers can see what the system was uncertain about
- Similar past decisions retrievable for comparison
- Access restricted appropriately given the data shown
- Explanation available without engineering involvement
Honesty about limits
Overclaiming explainability is its own risk.
- Model-generated rationales labelled as such, not as records
- Attribution and importance methods described as approximate
- Documented statement of what cannot be explained
- No claim that the system is fully interpretable when it is not
- Limitations communicated to reviewers who rely on explanations
- Uncertainty conveyed alongside the explanation
- Claims in marketing material checked against reality
Obligations and review
Check the requirements for your sector and jurisdictions.
- Decisions with significant individual effect identified
- Explanation obligations checked per jurisdiction
- Human review route available and staffed
- Review outcome recorded and fed back
- Timeframes for responding to a request defined
- Explanation quality sampled and reviewed
- Legal sign-off on the explanation approach
What are the most common failures?
Presenting a model's generated rationale as an explanation. Recording the conclusion without the inputs. Explanations that state a factor without enabling any action. And claiming interpretability the system does not have.
Who should own this?
The business owner of the decision owns the explanation; engineering provides the records; legal confirms the obligations. An explanation designed solely by engineering tends to describe the system rather than the decision.
How often should it run?
Designed at build time, reviewed when rules change, and sampled quarterly for quality. Any change to the decision logic requires the explanation to be re-checked.
What evidence should it produce?
Decision records retrievable by identifier, sample explanations reviewed for accuracy, and the documented statement of what can and cannot be explained.
What about systems where the model does decide?
Some cases genuinely need model judgement тАФ assessing tone, classifying ambiguous content, summarising. There the honest position is to explain the inputs and the process rather than the reasoning.
Say what the system was given, what it produced, and that a human can review it. That is true and useful. Presenting a generated rationale as the reason is neither. See why human oversight is a design problem.
What should you do first?
Take one recent automated decision and try to explain it from your records. Whatever you cannot reconstruct is what the logging is missing.
How FISTA Solutions helps
FISTA Solutions builds and operates production AI systems through AI agents, AI enablement, and forward deployed engineering: consequential decisions made by rules so the explanation is exact, with inputs and retrieved context recorded rather than reconstructed afterwards, decisions documented with their reasoning, and handover that leaves your team able to maintain what was delivered. The record is 150+ projects for 50+ companies across 12+ countries.
To adapt this checklist to your environment, message FISTA on WhatsApp, or read AI bias testing checklist.
Share-ready article cover
Download the generated social format.
Clear answers
Questions raised by this field note.
Straightforward guidance for evaluating scope, fit, and the next step.
01Can a model explain its own output?
It can generate text that reads like an explanation. That text is another output, not a record of the computation, and it can be confidently wrong about how the answer was reached.
02What is a real explanation then?
The inputs received, the context retrieved, the rules applied, and the deterministic logic that produced the outcome. Those are recorded facts rather than generated narrative.
03How do you make a decision explainable?
By making the decision itself deterministic. If a rule decides and the model only interprets the request, the explanation is the rule, which is both true and auditable.
04What do users need?
Why this outcome, what factors mattered, and what they could do differently. An explanation that does not enable any action is a disclosure rather than an explanation.
05Are there legal requirements?
In several jurisdictions, decisions with significant effect on individuals carry explanation and review rights. Check your obligations early. This is general guidance, not legal advice.
Continue exploring
Related capabilities
Start with the hard problem
Need the outcome owned, not merely analyzed?
Tell us where delivery is constrained. WeтАЩll map the fastest credible path from intent to verified production.