Regression testing for AI models and prompts
Catch recurring model failures when prompts, tools, or model versions change.
Turn important bugs into stable test cases
For each failure, record the input, the expected behavior, and the rule that defines acceptable output. An invoice assistant might be required to preserve the amount and currency even when it words the answer differently. Keep examples where a past fix made a measurable difference.
Set expectations for variation
Use exact assertions for structured fields and action permissions. Use a rubric where multiple phrasings are valid. For a nondeterministic model, repeat borderline cases and distinguish an occasional failure from a persistent regression. Record the model and prompt versions alongside the results.
Turn a failure into a reproducible case
An invoice assistant returns the amount correctly but changes the currency. Save a synthetic invoice, the expected amount and currency fields, and the accepted output schema. Keep separate examples for ambiguous symbols or multiple totals. This lets a regression suite tell the difference between a parsing defect and a case where the model should ask for clarification.
Separate requirements from acceptable variation
Assert exact equality for identifiers, structured values, and authorization outcomes. For prose, use a rubric focused on factual content and required omissions. Test any action with a controlled fixture and verify that the resulting state matches the request. When a response must preserve several requirements, keep individual assertions so a failure tells the team what broke.
Define a comparison process before release
Run the baseline and candidate on the same data with equivalent settings. For variable outputs, repeat borderline cases where a single run is inconclusive. Track both overall results and cases where the candidate introduces a new failure. If a critical check fails, record what happened, the condition that triggered it, and the change required before proceeding.
Keep the suite relevant as tasks change
Group tests by task and risk. Add cases for meaningful production errors while consolidating duplicates that test exactly the same behavior. Track whether each failure was fixed in the prompt, model, preprocessing, or authorization layer. When an application requirement changes deliberately, update the expected result with the decision and verify that the new expectation is backed by the current specification.