AAIA lesson 1: What AI auditing is

AI auditing is the independent assessment of whether an organisation’s AI systems are governed, built, operated and monitored in a way that supports its objectives and obligations. It differs from classic IT audit because the system’s behaviour is learned from data and is probabilistic, so the object under review is wider than the application. The ISACA Advanced in AI Audit (AAIA) exam has 90 multiple-choice questions in 150 minutes, split across three domains weighted 33%, 46% and 21%. This course follows one running case, a 400-person insurer, through most lessons.

Why classic IT audit is not enough

A conventional application does what its code says. An auditor can read the specification, inspect the logic, test a sample of transactions and expect the same input to produce the same output tomorrow. Control assurance rests on that determinism.

An AI system breaks three of those assumptions:

• Behaviour comes from data. The logic is not written down by a person; it is fitted to training data. The data is therefore part of the system and part of the audit scope.
• Outputs are probabilistic. The same question may return different answers, and “correct” is usually a rate measured against a threshold, not a yes or no.
• Behaviour can change without a release. A provider can update a hosted model, or the world can drift away from the training data, and nobody deploys anything.

The shape of the exam

The figure shows the whole course in one view. Domain 1, AI Governance and Risk (33%), covers models and requirements, governance, risk management, privacy and data governance, and ethics, regulation and standards. Domain 2, AI Operations (46%), covers data management, development lifecycle, change management, supervision, testing, threats and incident response. Domain 3, AI Auditing Tools and Techniques (21%), covers planning, testing and sampling, evidence, data quality and analytics, and reporting.

The largest domain is the one with the least traditional audit method in it. Studying audit technique hardest and skimming operations puts the effort into the smallest domain. ISACA also lists 23 supporting tasks that cut across the domains, such as evaluating ownership of AI risk, vendor programmes or the impact of AI on the workforce. They are a secondary classification, not a fourth domain, and the lessons teach them where they fit, for example workforce in lesson 2, configuration management in lesson 7 and identity and access in lesson 9.

Understanding the model before testing it (1A)

Subtopic 1A, AI Models, Considerations, and Requirements, calls for working literacy about models and about what must be decided before one is built: purpose, data, acceptable performance, constraints and the people affected. Three model families come up repeatedly.

Rule-based systems can be audited almost like code. Machine learning shifts the evidence towards data and metrics. Generative models are often hosted by a provider, so the auditor relies on contracts, documentation and the organisation’s own testing. The audit skills carry over from CISA-style work, as the AAIA versus CISA page explains.

Worked example: a claims triage model

The insurer introduces a model that ranks incoming motor claims for fraud review. Before any testing, a practitioner would want answers to a short list:

1. What is it for, and what is it explicitly not for? Intended use defines what “working” means.
2. What decision does it influence, and who is affected if it is wrong? A ranking that sends honest customers to a slow queue has a different impact than one that pays fraudulent claims.
3. What performance was required before release, and who set it?
4. Where did the training data come from, and does it resemble today’s claims?
5. Is the model built in-house, bought, or hosted by a provider, and who controls changes?

None of these needs the mathematics. They establish what exists, who owns it and what evidence should exist. If the insurer cannot state the intended use, the finding is a governance gap, not a technical one, and it is made before any accuracy figure is looked at.

Easy to confuse

• Training versus inference. Training is when the model is fitted; inference is when it produces outputs in use. Data risks concentrate at training, operational and misuse risks at inference.
• Model versus system. The model is one component; the system includes data pipelines, interfaces, people and decisions. Audit scope follows the system.
• Requirements versus performance. Requirements say what the system must do and how well (written before build); performance is what testing later measures. Without the first, the second has nothing to be compared with.

Lesson 2 turns to governance. The AI governance and risk topic page covers the frameworks that appear later, and ML basics, overfitting and metrics gives the vocabulary behind the metrics.

Key points

• AI auditing applies familiar audit method to systems whose behaviour is learned, probabilistic and able to change without a release.
• The exam has 90 questions in 150 minutes; the domains are weighted 33%, 46% and 21%.
• Data, provider and decisions are in scope, not only the deployed application.
• Rule-based, machine learning and generative models leave different evidence.
• Intended use and required performance must exist before deployment; their absence is a governance finding.

Check yourself

This lesson opens Domain 1 and frames the course; the exam-focused summary is on the governance and risk recap.