AAIA AI auditing tools and techniques
AI Auditing Tools and Techniques is worth 21% of AAIA — around 19 of the 90 questions, the smallest domain. If you hold CISA or an equivalent designation, this is the part you already know, applied to an unfamiliar subject.
Do not over-invest here. Do not skip it either: the application of familiar method to AI is where the marks are, and it differs from ordinary IS audit in specific ways.
Audit planning and design
Scoping an AI audit is harder than scoping an application audit because the system’s boundary is wider than the system:
- The model itself, and which version is in production
- The training data, including its sources
- The provider, if the model is hosted or pretrained elsewhere
- The decisions it influences and who they affect
- The oversight around it
A scope that covers only the deployed application misses most of the risk. Expect questions where the correct answer extends scope to data provenance or the third-party provider.
Testing and sampling methodologies
Traditional sampling assumes a population of comparable transactions. AI gives you a population of outputs, which behave differently:
- Outputs are probabilistic, so a single wrong answer is not automatically a control failure — you are assessing rates against thresholds.
- Sampling should be risk-weighted: edge cases, high-impact decisions and cases where the model was uncertain are worth more than random samples.
- Comparing across versions requires a fixed benchmark set, not fresh random samples, or you cannot separate model change from sample change.
Evidence collection techniques
The core intellectual work of the credential. Auditing a traditional control means inspection, re-performance or observation. For a model:
| You cannot | So instead you obtain |
|---|---|
| Read the weights and conclude anything | Validation and evaluation results against thresholds |
| Re-perform an inference to prove correctness | Benchmark results, dated and version-tagged |
| Inspect the logic | Documented model selection rationale and limitations |
| Trace a decision through code | Monitoring records and oversight review evidence |
| Confirm data by observation | Data lineage and source agreements |
The question to carry into the exam: what would sufficient, appropriate evidence look like for this assertion? If an answer option offers testimony where a record should exist, it is wrong.
Audit data quality and data analytics
Two directions, and the exam uses both:
- Auditing the data an AI system uses — completeness, accuracy, bias in the training set, labelling quality.
- Using analytics in the audit — analysing output populations at scale rather than sampling a handful, which is often the only practical way to assess a system making thousands of decisions.
AI audit outputs and reports
Reporting on a probabilistic system without overstating certainty:
- State what was tested, on which model version, and on what date. A finding about a model that has since been retrained needs that context to remain meaningful.
- Distinguish a control weakness from a performance observation. “Accuracy is 91%” is not a finding unless a threshold was defined and missed.
- Be explicit about limitations. If provenance records did not exist, you could not verify the data, and the report should say so rather than implying assurance you could not obtain.
Sample questions
Question 1. An auditor is scoping an engagement over a customer-facing AI assistant built on a hosted third-party model. Which scope decision is MOST appropriate?
- A. Limit scope to the application code and its deployment pipeline
- B. Limit scope to the user interface and accessibility
- C. Include the application, the data used for grounding, and the third-party provider arrangements
- D. Limit scope to the hosting infrastructure and network controls
Show answer
Answer: C
The risk extends beyond the deployed application to the data used for grounding and to the third-party provider whose model changes can alter behaviour, so scope must include both. Limiting scope to the application, the interface or the infrastructure leaves the principal sources of risk unexamined.
Question 2. An auditor wants to compare a model's performance before and after a version change. Which approach provides the MOST reliable comparison?
- A. Draw a fresh random sample of outputs from each period
- B. Run the same fixed benchmark set against both versions and compare results
- C. Obtain management’s summary of the improvement
- D. Review the vendor’s release notes for the new version
Show answer
Answer: B
A fixed benchmark set run against both versions isolates the effect of the model change, because the inputs are held constant. Fresh random samples confound model change with sample variation, management’s summary is testimony, and vendor release notes describe intent rather than measured outcome in your environment.
Question 3. An auditor could not obtain data lineage records for a model's training set. How should this be handled in the report?
- A. Disclose the scope limitation and state that provenance could not be verified
- B. Omit the area, since no evidence means no finding
- C. Accept management’s verbal assertion about the data sources
- D. Note it informally and defer the matter to the next audit cycle
Show answer
Answer: A
Where evidence could not be obtained, the report must disclose the limitation and the resulting inability to conclude, so readers do not infer assurance that was never established. Omitting it, relying on management’s assertion or deferring to the next cycle all imply a level of assurance the auditor does not have.
What to practise
Write an evidence request list for one AI system: ten items you would ask for on day one of the engagement. If your list is mostly documents and records rather than conversations, you have the instinct this domain rewards — and you will find the same instinct answers a good share of the operations domain too.