A model replacement can leave an AI workflow looking familiar while changing the decisions it makes. The interface still opens, the response still reads well, and the output may even arrive sooner. None of those observations establishes that the workflow continues to handle the same tasks within its agreed limits.
Regression testing asks a narrower question than a general model comparison: which behaviours that the business currently depends on would change? A useful release plan identifies those behaviours, runs the current and candidate configurations against comparable conditions, and gives an accountable person enough evidence to approve or reject the change.
Write a release contract before comparing answers
List what must remain true after the replacement. Separate business requirements from presentation preferences. Correct account selection, preserved uncertainty and valid tool arguments may be required. A particular opening sentence usually is not.
In a hypothetical Johannesburg maintenance business, an assistant converts incoming service requests into draft job records. Its contract might require it to preserve the reported fault, identify the supplied asset reference and leave an unknown appointment date empty. It must not promise attendance or create a confirmed booking.
Set explicit blocking conditions. A candidate that invents an appointment fails the release contract even if its summaries are clearer. Define acceptable limits for completion time, human correction and execution cost as local operating choices, not universal standards. Record who can accept a non-blocking weakness and what additional control they would require.
Freeze the comparison, including the surrounding workflow
Save a release manifest for the current configuration and the candidate. Include model identifier, request settings, prompt version, output schema, retrieval configuration and tool definitions. Record the test data version and the evaluation code version as well.
Change only the intended variable in the first comparison. If a new model requires a different adapter, document that difference; the tested release is then the model plus its adapter. Do not quietly rewrite prompts, refresh the knowledge base and switch models together, then attribute every improvement to the model.
Use fixed tool responses where practical for the initial comparison. Otherwise, a changed stock level or revised customer record can make two runs incomparable. Reset test state between runs so one execution cannot benefit from records created by the other.
Follow this controlled comparison with an integration run against a representative test environment. Fixed responses help isolate behaviour, but they do not test authentication, actual response shapes or unexpected service failures.
Select cases that can reveal a release-specific failure
Build the regression collection from the workflow's existing acceptance tests and past defects. Add cases that exercise boundaries likely to be affected by the proposed change: unusually long requests, multiple attachments, conflicting instructions, empty tool results and partial data.
For the hypothetical maintenance assistant, include a message containing two asset references but only one requested visit. Include a corrected address later in an email thread. Include a customer asking for an appointment that the business has not accepted. The expected behaviour should name the exact fact or boundary being tested.
Keep ordinary requests in the collection too. A candidate can perform well on difficult examples yet create friction in routine work. Group cases by operational importance and frequency, while showing rare high-impact cases separately rather than allowing their results to disappear into an average.
Protect a final acceptance collection from prompt tuning. If a failure becomes a development example, add a fresh variation for independent checking rather than pretending that repeated success on the familiar example proves broader reliability.
Compare decisions, records and execution traces
Run each case through both configurations and produce a paired record. Show required fields, forbidden actions, selected evidence and execution outcome. Classify the pair as unchanged pass, improvement, regression or shared failure. A defect in both versions remains a defect, even though the replacement did not introduce it.
Use deterministic checks for properties that have exact answers: permitted status values, identifier matches, missing-field handling and whether an unauthorised write occurred. Use reviewers for judgements such as whether a summary materially changes the customer's request. Avoid treating textual similarity to a preferred answer as the sole quality measure.
For action-taking workflows, inspect the resulting system state as well as the response. Anthropic's discussion of agent evaluations distinguishes grading outputs from checking an environment changed by an agent. Apply that distinction to the release: a reassuring message cannot compensate for a job saved against the wrong account.
Hide model identities during subjective review where feasible. Ask reviewers to explain disputed decisions against the contract, not vote for the answer that sounds more polished.
Repeat important cases and investigate disagreement
Choose a repeat policy before running the final suite. Apply the same policy to both configurations, and retain failed attempts rather than reporting only the best result. For consequential cases, examine whether the candidate consistently respects the boundary across repeats.
When runs disagree, inspect the earliest meaningful divergence. Did retrieval supply different evidence? Did the assistant choose a different tool? Did a valid tool result become an unsupported statement? Assign the defect to the component that needs changing instead of adding a broad prompt warning to every failure.
Keep the repair cycle honest. After changing a prompt or adapter, rerun the affected cases and a broader regression sample. A fix for missing appointment dates might accidentally reject requests that already contain an authorised booking reference. The original defect and the side effect both belong in the review record.
Include operating limits in the release decision
Measure the complete test execution, including retries, tool calls and time spent waiting for review. Use the same accounting boundary for both versions. A lower model-call cost is not enough if the candidate produces additional review work or repeatedly requests unnecessary information.
Exercise cancellation, timeout and malformed-response paths. Confirm that an interrupted attempt does not leave a draft marked complete or trigger a second action during retry. These checks concern the whole integration, even when the proposed release is described as a model swap.
Report the decision in plain terms: supported task groups, blocking failures, accepted limitations and unresolved questions. Link each regression to its evidence. Avoid a single aggregate grade that forces the process owner to infer whether a critical requirement passed.
Promote gradually with a tested return path
Before live routing changes, run the candidate in a mode that cannot write to production or send messages. If authorised shadow evaluation uses live inputs, preserve the same access restrictions and data-handling controls as the current workflow. Do not let both versions execute the same customer action.
For the pilot, define the eligible queue, accountable owner and stop conditions. Keep the previous deployment available, but test restoration rather than assuming it works. Consider records already created by the candidate: returning to an earlier model does not undo them.
Finish with a signed release record containing the comparison manifest, results and recovery procedure. The durable benefit of well-controlled AI integrations is not that every upgrade will pass. It is that the team can recognise a failed upgrade before it becomes normal operating behaviour.