AAISM lesson 7: AI threats and testing

AI threat management starts from one observation: an AI system has more places to attack than a conventional application, and each place needs a different control. Besides the code and the network, an attacker can reach the training data, the model itself, the prompt, the tools the model may call, and the output. The AAISM outline treats this as subtopic 2B, part of the 31% risk domain, and it also lists “identify and assess the AI threat landscape” and “design and implement testing and vulnerability management of AI solutions” among its supporting tasks.

One attack per stage

The figure walks through a system from left to right, and the same walk works in an exam scenario.

Training data: poisoning. An attacker who can influence data the model learns from can shift its behaviour, or plant a trigger that only fires on a chosen input (a backdoor). The control sits at the data pipeline: provenance, integrity checks, review of sources.

The model: extraction. Someone who can query a model many times can train a copy of it, or probe it to learn what it memorised. The visible signal is an unusual pattern of queries, so the controls are authentication, rate limiting and query monitoring.

The prompt: injection. An instruction-following model cannot reliably tell the operator’s instructions from text it was merely handed. Direct injection comes from the user; indirect injection hides in content the model later reads. No filter fixes this completely, so the design answer is to limit what a manipulated model could do.

Tools and plugins: excessive agency. When a model can send mail, query a database or call an API, the damage from a successful injection equals the permissions of those tools. Least privilege, narrow tool scopes and human approval for consequential actions apply here.

Output: data leakage and improper handling. The model may reveal sensitive information from training data or retrieved documents, and downstream systems may trust its output blindly. Output filtering, access-aware retrieval and treating model output as untrusted input are the answers.

A shared vocabulary

Two public catalogues help a security manager talk to testers and engineers without inventing terms. MITRE ATLAS is a knowledge base of adversary tactics and techniques against AI systems, modelled on the ATT&CK approach, and it includes case studies of real attacks. The OWASP Top 10 for LLM Applications (2025 edition) lists risks such as prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. NIST’s adversarial machine learning report (AI 100-2) organises the same ground by attacker goals and life cycle stage. The useful skill is naming the technique and its stage precisely.

Worked example

A 400-person insurer launches a claims assistant that reads uploaded documents and can open tickets in the claims system. A red-team exercise uploads a PDF containing hidden text telling the assistant to list the last ten claimants and put them in a ticket. The assistant complies.

A practitioner reads this as indirect prompt injection (stage: prompt), made harmful by excessive agency (stage: tools) and ending in data leakage (stage: output). Three controls follow: the ticket tool is limited to the current claim, uploaded content is treated as untrusted and kept apart from system instructions, and ticket creation that includes personal data needs approval. The finding is logged with an owner, severity and retest date, exactly like any other vulnerability.

Testing and vulnerability management for AI

Classic vulnerability management finds known flaws in known components, as described in the vulnerability management and CVSS topic. AI adds weaknesses that no scanner reports: behaviours. Red teaming means authorised people deliberately try to make the system misbehave, using the techniques above. Because behaviour is probabilistic, results are expressed as rates across many attempts.

Scanning of platform and libraries runs continuously; red teaming runs before release and after major change; evaluation suites rerun on every model or prompt change.

Managerially, the points are scope and rules of engagement, independence of the testers from the builders, a defined route for findings into the risk register, and retesting after fixes. See also common attack types.

Easy to confuse

• Poisoning vs evasion. Poisoning corrupts training; evasion crafts an input at inference time.
• Extraction vs inversion. Extraction copies functionality; inversion recovers information about training data.
• Prompt injection vs jailbreak. Injection smuggles instructions into content the model processes; a jailbreak talks the model past its own safety rules.

Key points

• Identify the stage first; the stage decides the control.
• Indirect prompt injection cannot be filtered out completely, so limit tool permissions.
• ATLAS and the OWASP LLM list give shared names for techniques and risks.
• Red teaming complements, and does not replace, scanning and evaluation.
• Findings need owners, severities and retests.

Check yourself

This lesson covers subtopic 2B of Domain 2, AI Risk Management (31%); the domain recap is at risk management explained.