What Is the Right to Explanation? AI Decisions and Accountability

Explainability is often approached as a technical problem — which attribution method to use — when the binding constraint is what the affected person needs in order to understand and contest a decision. Those are different requirements, and the second is better served by architecture than by interpretation techniques. This explainer covers both. It complements what is algorithmic transparency and ai governance framework, and reflects FISTA Solutions' approach in AI enablement delivery. This article is general guidance, not legal advice.

Where does the obligation apply?

Typically to decisions made solely by automated means producing legal or similarly significant effects: credit, employment, insurance, housing, benefits, education. The precise scope varies by jurisdiction and continues to develop.

Assistive systems, where a human genuinely makes the decision, are treated differently. That distinction carries weight only where the human involvement is substantive, which is a question about practice rather than about the diagram.

System type	Obligation strength	Design implication
Sole automated, significant effect	Strongest	Explanation required
Automated with rubber-stamp review	Effectively the same	Do not rely on the reviewer
Genuine human decision, AI assists	Lower	Record what was presented
Internal, no individual effect	Minimal	Ordinary documentation

What counts as an explanation?

Something the person can understand and act on. Which factors drove the outcome, in what direction, and what would need to change for a different result.

A description of the model, its training data, or its architecture is technically informative and useless to someone contesting a decision about their mortgage. The audience determines the content, and the audience is the affected individual rather than a regulator's technical team.

Are attribution methods enough?

They approximate. Feature attribution techniques estimate which inputs influenced an output under particular assumptions, and different methods can disagree about the same case. They are useful diagnostic tools for developers.

Presenting an approximation as the reason a decision was made is weaker than it appears, particularly if the person contests it and the method is examined. An account of what the system actually did is stronger than a reconstruction of what probably mattered.

What does designing for explanation mean?

Choosing an architecture whose decision basis can be recorded truthfully. Rules applied deterministically with model assistance on specific judgements. Structured criteria with the evidence for each recorded. Retrieval that cites the documents relied upon.

These produce genuine accounts, because the reason is captured at the time rather than inferred afterwards. The cost is architectural constraint; the benefit is that the explanation is real. See what is retrieval augmentation.

What must be recorded at decision time?

The inputs used, the version of the model and rules applied, the factors that drove the outcome, any human involvement and what that person saw, and the outcome itself. Reconstructing this later against a model that has since changed is not possible.

Retention needs to match the period during which a decision can be contested, which is frequently longer than ordinary log retention.

Does adding a human resolve it?

Only if the human genuinely decides. A reviewer processing forty recommendations an hour with an approval rate of ninety-eight percent is not meaningful involvement, and describing the system as human-decided on that basis is a position that will not hold.

Meaningful involvement means the reviewer has the information, the authority, the time, and demonstrably exercises judgement — which is measurable through override rates and review duration.

What should you do first?

Take one consequential decision your systems make and try to write the explanation a person would receive. If you cannot produce one from what the system records, that gap is the requirement, and it is an architecture question rather than an interpretability one.

How does this apply to generative systems?

Less directly and not at all cleanly. Most generative applications assist rather than decide, which places them outside the strictest obligations, and the boundary moves when an assistant's output is acted on without review. A drafting tool whose output is sent unedited has become part of the decision.

Where generative output feeds a consequential process, the defensible design records what the system produced, what evidence it drew on, and what the human changed. That record is what distinguishes assistance from automation in practice rather than in description.

What about contesting a decision?

An explanation that cannot support a challenge is not serving its purpose. The person needs enough to identify what they disagree with — a factor they believe is wrong, information they can correct — and a route to raise it that reaches someone who can act.

Designing the contest path alongside the explanation is what makes the whole arrangement meaningful, and it is the part most often left to a generic complaints process that has no visibility of how the decision was made.

How FISTA Solutions helps

FISTA Solutions designs consequential decision systems so the basis is recorded at the time, writes explanations for the affected individual rather than for engineers, avoids presenting attribution approximations as reasons, and measures whether human review is substantive rather than nominal, through AI enablement, AI agents, and forward deployed engineers. The record behind the approach is 150+ projects for 50+ companies across 12+ countries.

To build AI decisions you can actually explain, message FISTA on WhatsApp, or read ai governance framework.