Checklist · 4 minute read
AI Model Change Log Template: Recording What Moved and When
When behaviour changes unexpectedly, the change log is what makes investigation possible. Each entry should record what changed, why, the evaluation evidence, the rollout method, the cost and latency impact, and who approved it — including changes the provider made rather than you.
When behaviour changes unexpectedly, the change log is what turns a mystery into an investigation. This template covers what to record, drawn from FISTA Solutions' AI agents operational practice.
What does an entry record?
Six fields, all of which an investigation will want.
| Field | Why it is needed |
|---|---|
| What changed | The starting point of any investigation |
| Why | Whether the change is still justified |
| Evaluation evidence | Whether it was tested |
| Rollout method | Who was affected and when |
| Cost and latency impact | Whether economics moved |
| Approver | Accountability |
What changed
Specific enough to reproduce the previous state.
- Component identified: model, prompt, retrieval, corpus, tools, routing
- Previous value or version recorded
- New value or version recorded
- Scope: which systems and which users
- Timestamp of the change taking effect
- Link to the code or configuration change
- Whether the change is reversible and how
Reason
Why, in a sentence a stranger could understand.
- Problem or opportunity that motivated it
- Link to the ticket or incident that prompted it
- Alternatives considered where relevant
- Expected effect stated before deployment
- Whether it was forced by a provider deprecation
- Urgency and whether normal process was followed
- Any exception granted, with who granted it
Evidence
What was measured. See AI eval report template.
- Evaluation results linked
- Comparison against the previous configuration
- Regressions noted and their acceptance recorded
- Cost impact measured
- Latency impact measured
- Any testing skipped, stated explicitly
- Reviewer named
Rollout
Who was affected and when.
- Rollout method: staged, flagged, or full
- Percentages and dates per stage
- Watch period and what was watched
- Observed effect during the watch period
- Whether the rollout completed or was halted
- Flag state if the change is behind one
- Date the change reached everyone
Provider-side changes
The category most often missing.
- Provider announcements logged as entries
- Date the provider change took effect
- Evaluation run against the new provider behaviour
- Observed differences recorded
- Action taken in response
- Deprecation deadlines recorded with owners
- Unannounced changes logged when detected
Access and retention
Findable during an incident, at three in the morning.
- Entries searchable by date range
- Searchable by component
- Available during an outage of your primary systems
- Retained as long as the records they explain
- Linked from incident tooling
- Format consistent enough to scan quickly
- Owner named for the log itself
What are the most common failures?
Logging code changes only. Provider changes absent. Evidence linked from somewhere that later moves. Entries written after the fact. And a log stored where it is unavailable during an incident.
Who should own this?
Whoever makes a change writes its entry. A platform or operations function owns the log's format and availability.
How often should it run?
Per change. Reviewed during every incident and audited quarterly for completeness against deployment records.
What evidence should it produce?
The log itself, its completeness against deployment history, and its use during past incidents. A log nobody consulted during an investigation is not serving its purpose.
How does this relate to version logging in outputs?
They complement each other. Output logging says which versions produced a given result; the change log says why those versions were in place and what was tested.
Together they let an investigation move from a bad output to the decision that caused it. Either alone leaves a gap. See AI log retention checklist.
What should you do first?
Check whether your last provider model update appears anywhere in your change history. It usually does not, and that is the gap.
How FISTA Solutions helps
FISTA Solutions builds and operates production AI systems through AI agents, AI enablement, and forward deployed engineering: change entries written as part of making the change, covering provider-side shifts as well as your own, with evaluation evidence linked, decisions documented with their reasoning, and handover that leaves your team able to maintain what was delivered. The record is 150+ projects for 50+ companies across 12+ countries.
To adapt this checklist to your environment, message FISTA on WhatsApp, or read how to set up AI change control.
Share-ready article cover
Download the generated social format.
Clear answers
Questions raised by this field note.
Straightforward guidance for evaluating scope, fit, and the next step.
01What counts as a change?
Model version, prompt, retrieval configuration, corpus, tool definitions, and routing rules. Also provider-side changes you did not make but that altered behaviour.
02Why log provider changes?
Because they change your system's behaviour with no diff on your side. Without an entry, the shift looks inexplicable and investigation starts from nothing.
03What evidence should accompany an entry?
The evaluation run comparing new against previous, cost and latency measurements, and the rollout plan. Those are what a later investigation needs.
04How should entries be found?
By date range, so someone investigating a problem that started on a given day can see everything that changed around it. That is the primary access pattern.
05Who writes the entry?
Whoever makes the change, as part of making it. Entries written retrospectively are incomplete, and entries written by someone else are usually wrong about the reason.
Continue exploring
Related capabilities
Start with the hard problem
Need the outcome owned, not merely analyzed?
Tell us where delivery is constrained. We’ll map the fastest credible path from intent to verified production.