AAIA AI operations explained
AI Operations is worth 46% of AAIA — around 41 of the 90 questions, nearly half the exam and more than the other two domains combined. It is also the domain with the least audit method in it.
That combination is why candidates fail. An experienced auditor assumes an audit credential will reward audit knowledge, allocates study time accordingly, and meets a paper where almost half the questions are about how AI systems are built, changed, supervised and broken.
Seven areas, each with an audit angle.
Data management specific to AI
Training data is the source of behaviour, which makes it the highest-value audit target:
- Provenance — a defensible record of where every dataset came from
- Integrity — detecting modification
- Quality and labelling — labels are decisions, and bad labels become bad behaviour
- Segregation — separating data of different sensitivity
- Retention and disposal
Evidence to request: data lineage documentation, source agreements, quality assessment results, labelling procedures and reviewer records.
Development methodologies and lifecycle
How these systems get built: problem framing, data preparation, model selection, training, validation, deployment, monitoring. You need enough understanding to know what should happen at each stage and what record it should leave.
Evidence to request: stage gate approvals, model selection rationale, validation results, deployment authorisation.
Change management specific to AI
Named separately by ISACA, and the single most distinctive area in the domain. Traditional change management assumes behaviour changes when someone deploys something. For AI, behaviour changes three ways:
| Trigger | Visible in a change record? |
|---|---|
| Retraining on new data | Sometimes |
| A provider updating a hosted model | Usually not |
| Drift as the world diverges from training data | Never |
An organisation whose change process covers only the first has a gap. A scenario where a hosted model changed and nothing was recorded is a change management finding, not a vendor complaint.
Evidence to request: model version history, revalidation results per version, contractual change notification terms, drift monitoring output.
Supervision of AI solutions
Outputs, impacts and decisions. The audit question is whether oversight is meaningful or nominal:
- Who reviews, how often, and against what criteria?
- Do reviewers have authority to override — and is there evidence anyone ever has?
- Are impacts on affected people tracked?
A policy requiring human oversight proves intent. Records of reviews, including overrides, prove operation. The exam consistently rewards the second.
Testing techniques for AI solutions
You cannot re-perform your way to assurance over a model. Know what each technique demonstrates:
- Validation against a held-out set — performance on data not trained on
- Evaluation metrics — accuracy, and for generative systems, groundedness and relevance
- Benchmark or golden sets — a fixed set enabling comparison across versions
- Adversarial testing — behaviour under hostile input
- Fairness testing — outcome disparity across groups
Each answers a different question. A model can pass accuracy testing and fail fairness testing, and reporting only the first is a finding.
Threats and vulnerabilities specific to AI
Poisoning, backdoors, model extraction, inversion, evasion, prompt injection. Know which lifecycle stage each targets — training data, the model artefact, or inference — because the control and the evidence differ by stage.
Incident response specific to AI
An AI incident is often not an outage. Categories: harmful output, data leakage through responses, discovered poisoning, drift into unreliability, provider failure. Containment is awkward because behaviour lives in weights — you cannot patch a model the way you patch code, so containment usually means restricting or disabling capability while you investigate.
Evidence to request: an incident response plan that names AI-specific scenarios, and records of any exercises.
Sample questions
Question 1. An organisation retrains a production model monthly on fresh data. Change records exist for code deployments but not for retraining. Which is the BEST description of the weakness?
- A. Retraining monthly is too frequent for a production system
- B. Retraining changes behaviour but falls outside change management, so it occurs without assessment or revalidation
- C. Retraining consumes excessive storage
- D. The retraining scripts are not documented to the same standard as application code
Show answer
Answer: B
Retraining changes model behaviour, so excluding it from change management means behaviour changes in production without assessment, approval or revalidation. Storage consumption and documentation style are peripheral, and monthly retraining is not inherently too frequent.
Question 2. Management reports that a model passed all accuracy testing before release. The system makes decisions affecting individuals across several demographic groups. Which additional testing evidence should the auditor seek?
- A. Load testing results at peak volume
- B. Penetration testing of the hosting environment
- C. Usability testing with end users
- D. Fairness testing showing outcome disparity across the affected groups
Show answer
Answer: D
Accuracy measured overall can mask materially different outcomes between groups, so fairness testing across those groups is the missing evidence for a system affecting individuals. Load testing addresses performance, penetration testing addresses exploitability, and usability testing addresses experience.
Question 3. A generative assistant disclosed confidential information in a response to an external user. Which should the incident response plan require FIRST?
- A. Contain by restricting or disabling the affected capability
- B. Perform root cause analysis of the training data
- C. Notify the model provider and await their assessment
- D. Update the acceptable use policy to prevent recurrence
Show answer
Answer: A
Containment by restricting the affected capability stops ongoing disclosure while investigation proceeds, following the standard contain-then-investigate sequence. Root cause analysis, notifying the provider and updating the policy all follow containment.
What to practise
Take one real AI system and write, for each of the seven areas, the single piece of evidence you would request first. Seven lines. That exercise is the whole domain, it is exactly how the questions are framed, and it is the most efficient preparation available for nearly half the exam.