AAIR lesson 12: AI incidents and continuity

An AI incident is any event where an AI system causes or threatens harm, whether or not it is “down”: a chatbot that discloses customer data, a model that starts rejecting valid applications, a poisoned dataset found after training, or a provider withdrawing a model. Three linked pieces answer it: incident response to contain and learn, a business impact analysis (BIA) to know what matters most, and continuity and recovery plans (BCP and DRP) to keep the business running. Subtopic 3F in the ISACA AAIR outline names all of them, and an “Other Skills Tested” item asks for AI considerations in each.

Why AI incidents behave differently

Conventional incident handling assumes a system is working or broken. AI adds a third state: working and wrong. The service answers promptly while giving biased, fabricated or leaking output. Three consequences follow.

• Detection is harder. Uptime alerts do not fire; detection relies on quality metrics, KRIs, user complaints and sampled output review.
• Containment usually means restricting capability. A model cannot be patched like code. Typical steps: switch off a feature, tighten filters, route to human review, roll back to a previous model version, revoke an integration’s access.
• Root cause is murkier. Drift, a provider update, poisoned data, prompt injection or a design flaw may be behind it, and the evidence in data, logs and model versions must be preserved.

The usual phases still apply: preparation, detection and analysis, containment, eradication and recovery, and lessons learned (see the incident response lifecycle). Preparation matters most for AI: AI scenarios should be integrated into the enterprise incident management programme, sharing its severity scale, escalation paths and communication rules instead of living in a separate data science channel.

The figure above shows two tracks: incident response is reactive and event-driven; the BIA is proactive and feeds the plans for when an AI-dependent process cannot be delivered.

The BIA for AI-dependent processes

A BIA identifies critical processes, how fast their loss hurts, and what recovery they need. The usual measures apply: maximum tolerable disruption, recovery time objective (RTO, how fast service must return) and recovery point objective (RPO, tolerable data loss; see backup types, RTO and RPO). Two AI-specific additions:

• Impact includes decision quality, not just availability. A fraud model that is available but wrongly approving transactions can cost more than an hour of downtime.
• Dependencies must be traced: model, data feeds, provider, vector store, human review queue. The AI inventory (lesson 7) and third-party map (lesson 11) supply this.

BCP and DRP for AI

The two plans answer different questions, so keep them apart when reading scenarios.

AI-specific content: a manual or rules-based fallback per critical process, a trusted previous model version ready for rollback, backups of model artefacts, prompts, configurations and evaluation sets, and alternative providers. A fallback that has never run is an assumption, so test it; a tabletop exercise on an AI scenario is a cheap way to find gaps.

Worked example: the credit-scoring model

A 300-person lender uses a vendor-hosted model to pre-score loan applications. The BIA ranks the process as high criticality: brokers wait for answers within minutes, and wrong scores carry regulatory and customer harm.

One morning the vendor ships a model update and approval rates drop sharply. A KRI crosses its threshold and a high-severity incident is opened. Containment: scoring is suspended and applications go to the manual underwriting queue (the BCP fallback). The team pins the previous model version if the contract allows, or switches to a backup rules engine (the DRP route). Evidence is preserved, decisions made since the update are re-reviewed, and the vendor is asked for its change record. Afterwards the contract gains a change-notification clause and the recovery targets are reviewed. Recovery here includes correcting past decisions, not only restoring service.

Easy to confuse

• Incident response versus BCP/DRP: response handles the event; the plans keep or restore service. One incident can trigger all three.
• BIA versus risk assessment: the BIA measures the impact of losing a process; a risk assessment also weighs threats and likelihood.

Key points

• An AI incident may be harmful output, data leakage, poisoning, drift or provider failure, often with no outage.
• Containment usually restricts or rolls back capability instead of patching.
• The BIA for AI must weigh decision quality as well as availability, and trace dependencies.
• The BCP keeps the business operating; the DRP restores the technology.

Check yourself

This lesson covers subtopic 3F in Domain 3, AI Risk Program Management (42% of the exam); the programme management recap summarises it with the rest of the domain.