AAISM lesson 5: AI incident response

An AI incident is any event in which an AI system is compromised, misused or behaves in a way that harms the organisation or the people it serves, and it is handled with the familiar response cycle adapted to systems whose behaviour comes from data and weights. Subtopic 1E and the supporting tasks on investigation, handling and continuity ask the manager to prepare processes before anything happens. Two facts shape the topic: many AI incidents are not outages, and containment usually comes before root cause analysis.

What an AI incident looks like

Detection therefore relies on more than uptime monitoring. Output monitoring, user complaints, anomalous query patterns, and quality metrics against a baseline all serve as signals. See the incident response lifecycle for the general model.

The response flow

The figure shows the sequence, and each step has an AI-specific reading.

1. Detect. A signal arrives from monitoring, a user or a vendor.
2. Triage and escalate. Decide the severity, the affected systems and data, and who must be told. Pre-agreed criteria matter, because an AI incident can look like an ordinary bug.
3. Contain. Stop the harm first: restrict or disable the affected capability, revoke an agent’s credentials, switch to a fallback, block an input pattern. Preserve evidence such as prompts, outputs, logs and model versions.
4. Eradicate. Remove the cause: purge poisoned data, replace a tampered artefact, fix an overly permissive retrieval index, add input and output filters.
5. Recover. Restore service from a trusted version and verify behaviour before full return, which usually means re-running validation tests. “Roll back” is harder than for code because behaviour depends on data as well as weights, so known-good versions of both need to be retained.
6. Report and learn. Document the timeline and decisions, then update controls, playbooks and training.

Alongside this, notification follows regulatory and contractual duties. Privacy law, sector rules and customer contracts may require informing regulators or customers within set periods, and which applies depends on the facts. The manager’s role is to make sure legal and privacy are engaged early and that contact points and criteria are written down in advance, not to give legal conclusions.

Continuity for AI-dependent processes

Continuity planning asks what happens to a business process if its AI component is unavailable or untrustworthy. Questions include: Is there a manual or rules-based fallback, and have staff practised it? How long can the process tolerate the loss (see recovery objectives)? Are model artefacts, configuration, prompts and training data backed up and restorable? What if the provider fails, and is there a second source or an exit plan? Because a trustworthy-but-down system and an available-but-wrong system are both failures, plans also cover degraded modes, such as human review of every output until validation completes.

A worked example

A 400-person insurer runs a claims assistant grounded on a document index. A claimant reports seeing another customer’s claim details in a reply. Triage classifies this as high severity involving personal data. The assistant’s retrieval function is switched off at once and claims staff fall back to manual summaries, which is containment plus continuity. Logs are preserved. Investigation finds that a permission setting on the index was too broad, which is the cause to eradicate. The index is rebuilt with access checks, tested with queries from low-privilege accounts, and restored. Meanwhile legal and privacy determine whether notification duties apply, and the post-incident review adds retrieval-permission testing to the release gate.

Easy to confuse

• Containment versus eradication. Containment stops spread or harm; eradication removes the cause.
• Escalation versus notification. Escalation goes up inside the organisation; notification goes out to regulators, customers or partners.
• Disaster recovery versus continuity. Recovery restores systems; continuity keeps the business function going.

Key points

• AI incidents include exposure, manipulation, tampering, harmful output, drift and provider failure.
• Contain first and preserve evidence, then investigate and eradicate.
• Recovery needs trusted versions of models and data plus revalidation.
• Notification duties come from regulation and contracts; involve legal early.
• Continuity covers fallbacks, backups and provider exit.

Check yourself

This lesson covers subtopic 1E of Domain 1 (31%); the governance and programme management recap summarises the whole domain.