AAIA lesson 13: Worked scenarios

Scenario questions are solved by three reads: who is the professional (usually the auditor, sometimes an adviser), which stage the system is in, and what the qualifier asks for. The answer that follows is normally the action that produces evidence or reduces exposure in the right order, not the most thorough-sounding one. The scenarios below combine material from all three domains and apply the earlier lessons. This is a study method for working through questions, not a claim about how ISACA builds them.

Reading the scenario

1. Role. An auditor reports and recommends; management decides and implements. An answer in which the auditor fixes the control is usually wrong.
2. Stage. Is the system proposed, in development, live or being retired? Pre-deployment points to requirements, approval and validation; live points to monitoring and change; incident points to containment.
3. Qualifier. FIRST is about sequence, BEST about the strongest among valid options, MOST about weight or significance. Qualifiers can be combined with a role (“the auditor should”) and the stage to narrow the choice.
4. Evidence test. Prefer records and independent results to statements.
5. Scope test. If one option widens the view to data, provider or affected people, ask whether the scenario hints at that gap.

Scenario A: the hosted model nobody owns

The insurer lets several teams use a hosted language model through personal API keys to summarise customer letters. No central list of AI use exists. Internal audit is asked for a view.

• Role: auditor advising the audit committee. Stage: live, unmanaged.
• Domain 1 (governance): without an inventory and owners the organisation cannot say what is in use, so the first action is to establish what exists (interview, review of spend and network or key records), then ask for the AI policy and risk classification.
• Domain 2 (operations): the same finding exposes data leaving the organisation and no change notification from the provider.
• Domain 3 (audit): scope the engagement on a risk-ranked list once the inventory exists.
• First action: obtain or build the AI inventory with owners and purposes. Testing individual summaries before knowing the population would be premature.

Scenario B: strong accuracy, weak assurance

A lender’s pre-screening model reports 95% accuracy. Complaints suggest younger applicants are refused more often. The model is retrained quarterly.

• Role: auditor testing operation. Stage: live.
• Domain 2 (testing): accuracy does not show group-level outcomes, so request group error and approval rates with the fairness measure and its rationale.
• Domain 2 (change): confirm each retraining was approved and revalidated, and that monitoring covers drift.
• Domain 1 (ethics and regulation): relate findings to the organisation’s fairness commitments and to anti-discrimination obligations, noting that legal conclusions belong to counsel.
• Domain 3 (evidence): re-perform group metrics on a stratified extract reconciled to the decision log.
• First action: obtain group-level outcome evidence. Recommending retraining first would act before the problem is established.

Scenario C: an assistant leaks data

A customer chatbot with access to policy records returned another customer’s details after a crafted message. The service stayed up.

• Role: auditor reviewing incident handling after the event, or an adviser during it. Stage: incident.
• Domain 2 (incident): if the incident is live, containment comes first, for example restricting the lookup capability while logs and versions are preserved. If it is closed, review whether the sequence was followed and whether the cause (excess tool permissions) was fixed.
• Domain 1 (privacy): determine with the privacy function whether notification duties arise; the auditor checks that the assessment was made.
• Domain 3 (reporting): the report names the control weakness (least privilege, an incident plan with no AI scenarios) and states the version and date examined.
• First action depends on timing: contain if live; examine incident records if past. Reading the stage is the whole question.

Situation, domain, first action

Key points

• Identify role, stage and qualifier before reading the options.
• Auditors recommend and report; they do not run or fix the control.
• FIRST asks for sequence: inventory before testing, containment before root cause.
• Choose options that create dated, independent evidence over testimony.
• Widen scope to data, provider and affected people when the scenario hints at them.

Check yourself

Lesson 13 closes the course by combining all three domains; keep the cheat sheet beside the decision table when revising.