AAIR lesson 8: AI risk scenarios
An AI risk scenario is a short, testable story of how something could go wrong with an AI system, built from three ingredients: a threat that acts, a vulnerability that lets it succeed, and an asset that is harmed. Two facts shape everything that follows. First, AI systems are attacked at different points: training data, the model itself, and the live input channel. Second, many damaging scenarios involve no attacker at all, such as a model that quietly degrades or a staff member who pastes confidential text into a public tool.
The scenario chain
The figure below shows the chain a risk professional works along. A threat, a vulnerability and an asset combine into a scenario; the scenario is rated for likelihood and impact; the rating drives the treatment decision covered in the next lesson.
Working from the chain avoids a common error: listing frightening attack names without saying what is actually exposed. “Prompt injection” is a technique. “A customer-facing assistant that can read internal documents follows instructions hidden in an uploaded file and reveals pricing” is a scenario, because it names the vulnerability (the assistant treats untrusted text as instructions), the asset (pricing data) and a consequence.
Threats and attacks against AI systems
Public catalogues such as MITRE ATLAS and the OWASP Top 10 for LLM applications give a shared vocabulary for these attacks. They are references a practitioner uses; the published AAIR outline names no specific catalogue, so learn the concepts rather than memorising entries. The recurring ones:
| Attack or threat | What it targets | In plain terms |
|---|---|---|
| Data poisoning | Training or fine-tuning data | Tampered examples teach the model wrong or attacker-chosen behaviour |
| Prompt injection | The input channel of a language model | Text from a user or document is obeyed as if it were an instruction |
| Model theft (extraction) | The deployed model | Repeated queries or file access are used to copy its behaviour or weights |
| Evasion | Inference | Inputs crafted to be misclassified, such as altered images |
| Training-data leakage | Training data | The model reveals personal or confidential material it learned |
| Misuse | The system’s purpose | Users apply it for harmful ends, or staff use unapproved tools |
| Drift and hallucination | Output quality | No attacker; accuracy falls or plausible but false answers appear |
Asking which life cycle stage each threat touches tells the practitioner which controls and evidence to look for.
Likelihood, impact and the rating
Likelihood is how probable the scenario is within a stated period; impact is the consequence if it happens, expressed in terms the business cares about: money, harm to people, legal exposure, lost trust, interrupted operations. Combining them gives a risk rating. Rating the scenario before any controls gives inherent risk; rating it after controls gives residual risk.
Likelihood is hard to estimate for AI because techniques evolve and history is thin, and impact can exceed the system’s own cost, since one wrong automated decision may affect thousands of people. Record the assumptions behind each estimate so they can be challenged.
Qualitative or quantitative
Qualitative analysis uses scales such as low, medium and high, or a five-by-five matrix. It is fast and works when data is scarce, which is often true for AI. Quantitative analysis puts numbers on the scenario, for example an annual frequency and a loss range in currency, and suits decisions where cost has to be compared with a control. The topic page on qualitative and quantitative risk analysis covers the calculations.
Worked example: a 200-person logistics firm
A 200-person logistics company lets an AI assistant answer customer questions about shipments, drawing on a database of orders and contracts. The risk practitioner runs a short workshop with the product owner, security and legal.
- Asset: customer contract terms and shipment records.
- Threat: an external party sending crafted messages (prompt injection); separately, a careless employee pasting contract text into a public chatbot (misuse).
- Vulnerability: the assistant has broad read access and no check that its replies stay within the asking customer’s data.
- Scenario: “A crafted message makes the assistant disclose another customer’s contract rates, breaching confidentiality clauses.”
- Rating: likelihood medium (the interface is public and attempts are cheap), impact high (contractual and reputational). Rated high before controls.
The scenario is recorded with an owner and its assumptions, then passed to treatment. Drift as the product catalogue changes is rated as a separate scenario.
Easy to confuse
- Threat vs vulnerability: the threat is what can act (an attacker, a failure); the vulnerability is the weakness it exploits. Removing the weakness does not remove the threat.
- Poisoning vs prompt injection: the first corrupts what the model learns; the second manipulates what it does at run time.
- Assessment vs testing: a threat assessment reasons about what could happen; adversarial testing, sometimes called red teaming, produces evidence about what does happen. They complement each other.
Key points
- A scenario names a threat, a vulnerability and an asset, and states a consequence.
- AI attacks target the training data, the model or the live input; many scenarios involve no attacker.
- Likelihood and impact give a rating; the same rating before and after controls gives inherent and residual risk.
- Qualitative analysis suits scarce data; quantitative analysis suits cost-based decisions.
- Record assumptions and review ratings because AI risk shifts over time.
Check yourself
Question 1. A company fine-tunes a classifier on examples scraped from a public forum. A risk practitioner worries that an outsider has seeded the forum with deliberately misleading examples. Which threat is described?
- A. Prompt injection
- B. Model extraction
- C. Data poisoning
- D. Model drift
Show answer
Answer: C
Misleading examples placed to alter what the model learns are data poisoning. Prompt injection acts at run time through input, model extraction copies a deployed model, and drift is a gradual change with no deliberate actor.
Want more questions like this? Full AAIR practice tests →
Question 2. A workshop lists the phrase 'attackers may exploit our chatbot' as a risk. Which change MOST improves it as a risk scenario?
- A. State the asset exposed, the weakness exploited and the resulting consequence
- B. Colour the entry red in the register
- C. Name the security product that will be bought
- D. Ask the chatbot vendor to rate it
Show answer
Answer: A
A usable scenario names the asset, the weakness and the consequence so it can be rated. Adding a colour, naming a tool or deferring to a vendor does not make the event assessable.
Want more questions like this? Full AAIR practice tests →
Question 3. A team has almost no incident history for a new AI tool and must prioritise ten scenarios by next week. Which approach is MOST appropriate?
- A. Quantify every scenario with annual loss figures
- B. Rank scenarios by the purchase cost of the tool
- C. Wait until incidents provide enough history
- D. Use a consistent qualitative scale, then quantify only the top scenarios
Show answer
Answer: D
With scarce data and little time, a qualitative scale applied consistently is quick and defensible, and the most significant scenarios can be quantified later. Full quantification needs data the team lacks, ranking only by cost of the tool ignores impact, and waiting for incident history leaves risks unassessed.
Want more questions like this? Full AAIR practice tests →
This lesson opens Domain 3, AI Risk Program Management (42%), and covers subtopic 3A; the recap page AI risk programme management places it within the whole domain.