AAIA lesson 12: Audit analytics and reports

Audit conclusions are only as good as the data behind them, and when the auditor uses analytics or an AI tool, that tool becomes one more thing to validate. Reports on AI systems must also state versions, dates and limitations because the subject is probabilistic and changes. This lesson covers subtopics 3D (Audit Data Quality and Data Analytics) and 3E (AI Audit Outputs and Reports), plus the two supporting tasks about using AI inside the audit process: evaluating the impacts, opportunities and risks of doing so, and using AI tools to improve planning, execution and reporting.

Data quality for audit purposes

Before running a test on a log, extract or dataset, the auditor checks that it can be relied on:

An extract prepared by the area under review, with no way to confirm the query used, is weak evidence however clean it looks. Document source, query, extraction date and reconciliation so the work can be repeated.

Analytics in AI audits

Analytics lets an auditor examine a whole population instead of a sample. Typical uses on AI systems: distribution of model scores and decisions over time, approval or error rates by group and version, override rates by reviewer, outliers such as unusually fast reviews, and gaps in logs. Analytics supports sampling: use it to find where the risk is, then examine selected items in detail. Results are leads until validated; an anomaly is a question, not a finding.

AI-enabled audit tools and their risks

Auditors increasingly use AI to draft working papers, summarise documents, classify evidence or flag anomalies. The supporting tasks in the outline ask for evaluation of both the opportunity and the risk. A practical control set:

• Accuracy: AI summaries and extractions can be wrong or invented; verify against the source document before relying on them.
• Confidentiality: audit data pasted into an external tool may leave the organisation; use approved tools with appropriate data terms.
• Accountability: responsibility for conclusions stays with the auditor, who must be able to explain how evidence was obtained.
• Documentation: record which tool, version and prompts were used, what was verified, and by whom.

Test the tool on known cases before relying on it.

Findings, ratings and recommendations

A finding normally has five parts: criteria (what should be), condition (what was found, with the version and date), cause (ideally the root cause, discussed with management), effect or risk, and a recommendation with an owner and timeline. A difference between criteria and condition is only a potential finding until its significance is judged from the likelihood of the risk and its impact. Ratings reflect that significance, not effort. A common scheme runs highest, significant, moderate, low and administrative, though organisations name the levels differently. Highest-rated examples are a system affecting individuals with no owner or oversight; administrative ones are naming inconsistency in documents. Working papers should let another competent auditor repeat the work and reach the same result.

AI-specific reporting points:

• State the model version, dataset and date covered; behaviour may differ after retraining.
• Separate a control weakness from a performance observation. “Accuracy is 91%” is not a finding unless a threshold was set and missed.
• Disclose limitations. If provenance records or provider details could not be obtained, say that assurance was not possible rather than implying it.

Worked example

The insurer’s audit of its claims triage model finds that retraining occurs quarterly without approval, and group-level monitoring has been absent for a year. Criteria: the insurer’s change policy and its own commitment to fairness monitoring. Cause: retraining is run by the analytics team, outside the change process. Effect: behaviour changes with no approval or revalidation, and unfair outcomes could go unseen. Rating: significant on the scheme above. Recommendation: bring retraining into change management and start monthly group metrics, owned by the head of claims, within 90 days. The audit committee summary opens with the conclusion and rating, lists the two themes, notes that provider internals were out of reach and describes follow-up. Follow-up then tests whether the actions exist and operate, not just that a plan was written.

Easy to confuse

• Finding vs observation: a finding deviates from a criterion; an observation may be useful context without one.
• Analytics result vs evidence: a flagged anomaly needs validation before it supports a conclusion.

Key points

• Check completeness, accuracy, timeliness and provenance of audit data before relying on it.
• Analytics covers entire populations but its outputs are leads until validated.
• AI audit tools need verification, data protection, documentation and retained auditor accountability.
• Findings state criteria, condition, cause, effect and a recommendation with owner and date.
• AI reports must identify versions, dates and limitations.

Check yourself

The domain recap is on the audit tools and techniques page.