AAIA lesson 10: Audit planning and design
Planning an AI audit follows the same stages as any audit (plan and scope, design the programme, test, analyse, report, follow up) but the scope is wider than the system and the skills needed are rarer. A risk-based plan for an AI solution reaches the model version in production, the training and grounding data, the provider, the decisions influenced and the oversight around them. This is subtopic 3A, Audit Planning and Design, the first of Domain 3 (AI Auditing Tools and Techniques). The figure shows where planning sits in the flow.
What changes when the subject is AI
Standard audit practice (as set out in ISACA’s ITAF and the IIA’s Global Internal Audit Standards) still applies: understand the activity under review, assess its risks, set objectives and scope, identify evaluation criteria, plan resources, and write a work programme that lists the criteria, tasks, methods and who does what. Four AI-specific pressures reshape it:
- Boundary: the risk lives in data, models and third parties, not only in the application.
- Opacity: internal logic cannot be inspected, so evidence shifts to records, validation results and monitoring.
- Pace: versions change, so the audit must fix the version and date it covers.
- Impact: outputs may affect people, so ethics, fairness and legal exposure join the risk picture.
Risk-based scoping
Start from the inventory of AI systems (lesson 2) and rank them. Higher priority goes to systems that make or strongly influence decisions about people, run autonomously, use personal or sensitive data, are customer-facing, rely on an external provider or are poorly documented. Management’s own risk classification is an input to be tested.
The ranked inventory feeds the audit universe, the inventory of auditable areas from which the audit plan is drawn; AI systems and AI processes belong in it. Then choose the form of the engagement: a single-system audit, a thematic audit (for instance, change management across all AI systems) or an audit of the governance programme. The scope statement should name the system and version, data sources, lifecycle stages, the period, and the exclusions with reasons.
Objectives and criteria
An objective states what the audit will conclude about, such as whether decisions made by the pre-screening model are subject to effective oversight. Criteria are the benchmark: the organisation’s own AI policy and risk appetite, applicable law and regulation, contractual terms, and frameworks or standards the organisation has adopted, for instance the NIST AI RMF as guidance or ISO/IEC 42001 as a management system standard. An auditor can only report a deviation against a stated criterion. If the organisation has no AI policy, say so and agree reference criteria with the audit sponsor beforehand, since “good practice” is otherwise unmeasurable.
Resources, skills and specialists
AI audits often need knowledge in data science, security, privacy, law and the business domain. The auditor does not need to build models but must understand them well enough to scope, challenge and judge evidence. Using a specialist is normal practice and brings three obligations: define the task and deliverables in writing, assess the specialist’s competence and independence, and remain accountable for the conclusion. A specialist’s opinion is evidence the auditor evaluates, not a replacement for judgement. Plan access early, because much of the evidence sits with a third party.
Worked example
The insurer asks internal audit to review its claims-triage model, built by a vendor and retrained quarterly. The auditor ranks it high: it influences payouts, uses personal data and depends on a supplier. Scope: the production version at a stated date, the training and retraining data, the vendor arrangement, change control over retraining, and the adjuster review step. Criteria: the insurer’s AI policy, its data protection obligations, and its model risk standard. Resources: two auditors, a data specialist for re-performance and legal input on privacy.
The audit programme then maps each risk to a test:
| Risk | Control expected | Test | Evidence |
|---|---|---|---|
| Unfit model | Pre-release validation | Inspect results vs thresholds | Dated validation report |
| Uncontrolled retraining | Change approval | Trace each retrain to approval | Change records, version log |
| Unfair outcomes | Fairness monitoring | Re-perform group metrics on a sample | Monitoring output, extract |
| Vendor change | Contract and notice | Inspect contract, notices | Agreement, correspondence |
| Nominal oversight | Adjuster review | Sample cases, check overrides | Review logs |
Design and operating effectiveness
A supporting task asks for evaluation of the design and effectiveness of controls specific to AI. Design effectiveness asks whether a control, if it works as described, would address the risk: a validation requirement with no threshold or independent reviewer is poorly designed. Operating effectiveness asks whether it worked in the period: validation reports exist for each release, approvals precede deployment. Design is assessed first, because testing the operation of a control that cannot meet the risk wastes effort.
Easy to confuse
- Design vs operating effectiveness: a well-designed control may not operate; a control that operates may not be well designed.
- Scope vs objective: scope is what is examined; the objective is the conclusion sought.
- Criteria vs evidence: criteria are the standard for comparison; evidence shows what actually happened.
- Specialist vs auditor: the specialist informs; the auditor concludes.
Key points
- AI audit scope extends to data, model version, provider, decisions and oversight, not just the application.
- Risk ranking draws on impact on people, autonomy, data sensitivity and third-party dependence.
- Criteria must be agreed before fieldwork, even if the organisation lacks a policy.
- Specialists are used under written terms, with competence assessed and accountability retained.
- An audit programme links each risk to a control, a test and the evidence expected.
Check yourself
Question 1. An auditor is scoping a review of a third-party credit model used by the organisation. Which scope is MOST appropriate?
- A. The model version in use, its input data, the supplier arrangement and decision oversight
- B. The user interface and its accessibility
- C. The network connections to the vendor
- D. The vendor’s marketing claims about accuracy
Show answer
Answer: A
Risk arises from the model version in use, the data feeding it, the supplier arrangement and the oversight of decisions, so scope should include all of them. Limiting scope to the interface, the network or the vendor’s marketing material leaves the main risks unexamined.
Want more questions like this? Full AAIA practice tests →
Question 2. The organisation has no AI policy. What should the auditor do FIRST when defining audit criteria?
- A. Proceed with fieldwork and define criteria from the results
- B. Apply a peer organisation’s published AI policy without telling management
- C. Agree reference criteria, such as law and an adopted framework, with the audit sponsor
- D. Postpone the audit until the organisation has written a policy
Show answer
Answer: C
Criteria must be agreed before fieldwork so that deviations can be reported against something. Defining them afterwards, imposing another organisation’s policy without agreement, or cancelling the audit does not give a basis for conclusions; agreeing reference criteria with the sponsor does.
Want more questions like this? Full AAIA practice tests →
Question 3. An audit team lacks data science skills for a model review. Which approach is MOST appropriate?
- A. Exclude the model from scope without disclosure
- B. Rely on the model owner’s opinion without challenge
- C. Let the specialist issue the audit conclusion
- D. Engage a specialist under defined terms and evaluate their work while retaining accountability
Show answer
Answer: D
A specialist can be used if the task is defined, competence and independence are assessed and the auditor retains accountability for the conclusion. Dropping the area, relying on the model owner or delegating the conclusion all weaken the audit.
Want more questions like this? Full AAIA practice tests →
The domain recap is on the audit tools and techniques page.