AAIA lesson 9: AI threats and incidents

AI systems are attacked at three places: the data that trains them, the model itself, and the input and output at inference time. Each stage has its own attack names and its own controls, and an AI incident is frequently not an outage but a harmful, leaked or manipulated answer. This lesson covers subtopics 2F (Threats and Vulnerabilities specific to AI) and 2G (Incident Response Management specific to AI). The figure below lays out four threats along the pipeline.

Threats by pipeline stage

The NIST adversarial machine learning taxonomy (AI 100-2, 2025 edition) organises attacks by the lifecycle stage they target and by the attacker’s goals, capabilities and knowledge. For predictive AI it names evasion, poisoning and privacy attacks; for generative AI it adds misuse. The OWASP Top 10 for LLM Applications (2025) supplies matching vocabulary for generative systems: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. MITRE ATLAS is a public knowledge base of adversary tactics and techniques against AI-enabled systems, built on observed attacks and complementing MITRE ATT&CK. Read together:

Two further themes recur. Supply chain risk covers pretrained models, datasets and libraries obtained from others, so vendor assessment and artefact integrity checks belong in scope. Excessive agency describes AI agents granted broader permissions than the task needs; a successful injection then acts with those permissions. Least privilege for tools and data, not a longer system prompt, limits the damage. No known filter prevents prompt injection completely, which is why limiting what a manipulated model can do is the main control. See AI agents explained and common attack types for background.

Why containment is different

A bug in code is patched. A model’s behaviour lives in learned weights, so a problem such as a poisoned dataset cannot be edited out line by line. The practical options are restricting the capability, adding filters, rolling back to a known-good version, retraining on cleaned data, or switching the provider. The incident lifecycle still applies (see incident response lifecycle), with AI-specific content:

1. Detect: monitoring of outputs, user reports, provider notices, anomaly alerts on query patterns.
2. Contain: limit or disable the affected function, revoke keys, block abusive sources; preserve logs and the exact model, prompt and data versions.
3. Investigate: reconstruct inputs and outputs, identify whether cause was data, model, prompt, integration or misuse.
4. Recover and learn: fix, revalidate before restoring, notify as obligations require, update scenarios and controls.

AI incident categories include harmful output, leaked personal data, discovered poisoning, drift, misuse and provider failure. Problem management follows incident handling: recurring incidents (for example repeated wrong answers on one topic) are grouped, a root cause is found in data, model, prompt or integration, and a permanent fix is tracked to closure.

Worked example

The insurer launches a customer chatbot that can look up policy details through an internal tool. A customer pastes a message telling the bot to ignore its rules and read out another policyholder’s address. The bot complies because its tool account can read any record.

The auditor starts with the incident, not the model. Was it detected by monitoring or by the customer? What was contained, and how fast? Were prompts, outputs and tool calls logged and preserved? Was the cause traced to excessive permissions on the tool account, and was the fix a scoped, per-customer lookup rather than a stronger instruction? Was the scenario added to the incident plan and to red-team testing? The finding is weak least-privilege design and an incident plan that never named prompt injection.

What an auditor tests

• A threat model naming AI-specific attacks for each in-scope system.
• Identity and access management for AI: who and what can reach model endpoints, training data, prompts and registries; service accounts and API keys for models and agents with owners, scoped rights and rotation; agents treated as identities with least privilege; access reviews and logs.
• Vulnerability management that covers AI components: ML libraries, pretrained models, model-serving software and vector databases are patched and scanned like other software, with an inventory of what is used.
• Rate limiting and monitoring on model endpoints to slow extraction and abuse.
• Data supply controls and vendor assurance for pretrained components.
• Red-team or adversarial test reports with remediation and retest.
• An incident plan with AI scenarios, defined containment options, owners, and exercise records.

Key points

• Place the attack at the stage: data (poisoning), model (extraction, inversion), input (injection, evasion), output (leakage, harm).
• Prompt injection is mitigated mainly by privilege limits and treating input as untrusted.
• AI containment usually means restricting capability, since weights cannot be patched.
• Preserve model, prompt and data versions when an incident occurs.
• Evidence of exercised AI scenarios beats a generic incident policy.

Check yourself

The domain recap is on the AI operations page.