An AI workflow can behave differently after a release even when its visible interface has not changed. The team may have edited an instruction, replaced the model, refreshed the document collection or adjusted which search results reach the answer generator. A note saying "improved responses" is not enough to explain the difference.
A useful change log connects an observed problem to a specific intervention, its test evidence and the route back to an acceptable configuration. It should help an operator answer what changed and why, without requiring a reconstruction from chat messages and deployment history.
Log the deployed combination
Give each deployed workflow configuration a release identifier. Associate it with the exact prompt revision, requested model identifier, relevant generation settings, retrieval configuration, document collection version, output schema and tool definitions. Include the application revision that connects these parts.
Store substantial artefacts in their normal controlled repositories and link to them. The change log should be an index to evidence, not a second copy of every prompt and document. A prompt hash without access to the corresponding text is insufficient for investigation; a pasted prompt without a stable revision is equally awkward to maintain.
Distinguish what was requested from what the environment reports actually ran. Where a supplier does not expose a precise underlying version, record that limitation. The release record should describe the evidence available, not imply reproducibility that the team cannot demonstrate.
Begin each entry with an observable problem
Write the reason for the change in terms a reviewer can test. "Make the assistant smarter" gives nobody a stopping condition. "The assistant omits the effective date when summarising an amended procedure" identifies both a failure and a candidate acceptance check.
Record the affected task type and the known boundaries of the problem. An issue observed in short policy extracts does not automatically justify changing every workflow that shares the same model. If the cause remains uncertain, label the change as an experiment rather than presenting the diagnosis as settled.
Name an owner for the decision and an owner for operational follow-up. They may be the same person in a small team. What matters is that someone can explain the trade-off and someone will inspect the first live results after deployment.
Show the behavioural difference, not only the edit
A text diff explains which words changed. Add a concise account of the expected behaviour: which answers should become different, which must remain unchanged and which new refusal or escalation conditions are intended.
Consider a hypothetical internal maintenance assistant. Its current prompt asks for a concise answer; a proposed revision additionally requires the effective date and a source reference whenever it describes a procedure. The log should state that longer answers are an accepted trade-off, while unsupported dates remain unacceptable.
Record retrieval changes separately. In this hypothetical release, suppose the team also changes the filter to exclude documents marked as retired. That is a distinct intervention with distinct tests. A better answer cannot automatically be attributed to the prompt if the source collection changed at the same time.
Where several changes must ship together, explain the dependency. Otherwise, separate them enough to compare outcomes. The aim is understandable evidence, not an absolute rule that every release contains only one edited value.
Keep evaluation evidence comparable
Link the release entry to the test set revision, inputs, expected properties and reviewer decisions. Preserve outputs from both the current and candidate configurations. Summaries of results should distinguish improved cases, unchanged cases and regressions.
For the hypothetical maintenance assistant, test a current procedure, a retired procedure, two conflicting revisions and a question for which no approved procedure exists. Check whether the date and citation are supported, not merely present. The no-answer case matters because the new formatting requirement must not encourage a fabricated reference.
Record operational observations alongside answer quality: completion failures, retries, response time and reviewer effort. Use your own measured results rather than borrowing a benchmark to describe your workflow. If a test was skipped or a sample was too narrow, state that plainly in the acceptance note.
Do not overwrite disappointing trial results with the final successful run. Retaining the sequence helps explain why the accepted settings differ from the original proposal and prevents an abandoned experiment from being mistaken for a production configuration.
Make rollback an executable decision
A rollback note should name the previous configuration, who may restore it, what conditions justify that action and how restoration will be checked. Avoid instructions that amount to "put it back" without identifying the relevant artefacts.
Check compatibility before release. If the new output includes extra fields, can the old application still consume saved records? If a document collection has been rebuilt, is the previous collection available under the organisation's retention rules? A configuration reference is useful only when the components it names can still be used.
Separate restoration from repair. Reverting a prompt does not by itself correct summaries already saved or messages already sent. Include a way to identify work processed during the affected interval and route any required corrections to the responsible team.
Practise a rollback in the test environment. Record what was actually restored and how a sample task was checked afterwards. That exercise can reveal missing dependencies before an incident makes the same questions urgent.
Keep the entry short enough to maintain
Use a consistent set of fields, but allow the explanation to match the change. A spelling correction with no intended behavioural effect does not need the same discussion as a new retrieval policy.
A workable entry contains:
- Release identifier, deployment time and affected workflow.
- Problem statement, intended behaviour and accountable owner.
- Previous and new artefact references, including data dependencies.
- Evaluation evidence, accepted trade-offs and unresolved limitations.
- Rollback trigger, restoration target and verification task.
- Follow-up date and the decision made after observing live work.
Generate identifiers and configuration references from the release process where possible. Ask people to write the judgement that cannot be generated: why this change is worth making and which risks they accepted. Do not copy customer inputs, credentials or unrestricted document content into a broadly accessible release log.
Close the loop after deployment
Mark a release as deployed only when the target environment has been checked. Record partial rollout states explicitly so support staff can tell which team or task used which configuration.
At the agreed review point, compare live examples with the expected behaviour and document whether the release stays, changes or rolls back. Keep unexpected observations attached to the release even when they produce a separate issue.
An operational dashboard can group errors and review outcomes by release identifier, while the change log supplies the explanation behind those patterns. The two should complement each other. Start by documenting the current configuration and the next real change; a complete historical reconstruction is less useful than making tomorrow's investigation straightforward.