← All articles
AI security

AI model poisoning and how to detect and contain it

An in-depth guide to model poisoning, training data integrity, checkpoint and adapter provenance, focused evaluation, and recovery.

What model poisoning means

A poisoning incident involves deliberate interference with the integrity of a learned model. The interference can happen through data used to train or fine-tune it, through updates a training system accepts, or by tampering with the model artifact that reaches deployment. A replacement checkpoint can look like a normal version update because the application still loads successfully and answers most requests. An unexpected answer is a reason to investigate. The evidence for poisoning comes from tracing the change to model behavior back to a compromised input, update, or artifact.

Distinguish a changed model from changed inputs

A support assistant can start giving incorrect policy answers because it received a new model, a different adapter, a stale document index, or a changed system prompt. Training data poisoning affects what the model learns. A corrupted retrieval index can change what the model sees at inference time while its weights remain identical. Prompt injection is also an inference-time concern when untrusted content tries to steer a response or action. Preserve the deployed model and configuration identifiers before debugging so each layer can be compared against a known-good release.

Audit every artifact that can change model behavior

For a self-hosted model, inventory the base checkpoint, fine-tuning adapters, tokenizer, and any model files used by the serving process. Record their exact versions and content digests. Tie each artifact to an approved publisher or build identity and verify the signature or provenance against that expectation before promotion. A digest checks whether the bytes match an expected artifact. The trust decision depends on how the expected digest was established and who approved it. For managed APIs that do not expose weights, record the available model version, deployment configuration, change notices, and observable behavior baselines.

Keep provenance for annotation and fine-tuning data

A useful dataset manifest records the dataset version, source and batch identifiers, labeling guideline version, transformations, and approvals. Track relabeled items and late additions as explicit changes. Before training, compare the candidate dataset with the last approved version by source, label, and volume. An unexpected shift can have an ordinary cause, such as additional examples from a new collection partner. Quarantine batches with unexplained changes, review original examples in context, and record why each batch was cleared or rejected.

Measure failures where the consequences differ

Use a broad benchmark to catch general regressions and smaller, independent sets for decisions where failure matters. In a synthetic support-routing example, an approved model gets 470 of 500 routine cases right and a candidate gets 472. The approved model also escalates 39 of 40 cases that require human review, while the candidate escalates only 28. Report those counts separately. The escalation drop can justify holding the release even though the routine score improved slightly. It indicates a real behavior change in the test, while its cause still needs investigation.

Make release checks reproducible

Run the candidate and the baseline on the same evaluation records, scoring rules, and serving configuration. For outputs that vary between runs, repeat ambiguous cases and record the settings used. Reserve a protected evaluation set that is separate from fine-tuning and ordinary development feedback. Define release gates in advance for behaviors such as required escalation and sensitive actions. Keep the raw counts and representative examples so another reviewer can reproduce the comparison and decide whether a failure is caused by the model or the surrounding application.

Investigate suspicious behavior one layer at a time

Consider a document assistant that first shows a serious regression after a release combining a new adapter and a re-indexed knowledge base. Pause further promotion and preserve the release manifest, candidate artifacts, and authorized audit records. Compare the approved and candidate builds in an isolated evaluation environment with fixed prompts and fixed retrieval results. Then hold the model constant and compare the previous and current indexes. If the behavior follows the adapter, review its provenance and training inputs. If it follows retrieval, examine document provenance and index changes. Treat each result as evidence about the affected layer.

Contain the incident and verify recovery

Stop serving the suspect release if its behavior creates material risk and restore a previously approved version from trusted storage. Keep the suspect artifact isolated for authorized investigation. If publishing or storage credentials were compromised, revoke them at the issuer and review changes made during the time window. Re-run the same critical evaluation cases after rollback. Before promoting a replacement, verify its inputs and provenance, record the decision, and monitor the behavior that prompted the investigation.

Assign responsibility at each handoff

The data owner should be able to explain what changed between dataset versions. The training or release owner should be able to identify every model artifact in production and show who approved it. The evaluation owner should be able to run the critical checks against a rollback candidate. Rehearse that handoff using an approved test release with a deliberately missing attestation or a known benign regression. Confirm that the release is held, the right person is notified, and the previously approved model can be restored.