AAIA lesson 3: AI risk, privacy, data

AI risk management applies the usual cycle of identify, assess, treat, monitor, but with measurable thresholds and a risk profile that can shift without any change being made. Privacy and data governance then govern the raw material: what data may be used, for which purpose and for how long. This lesson covers subtopics 1C (AI Risk Management) and 1D (Privacy and Data Governance Programs); supporting tasks ask the auditor to evaluate the data governance and privacy programmes specifically for AI, and the effects of AI on people, other systems and the environment.

The risk cycle with AI-specific content

The method is the one already covered in qualitative versus quantitative risk management; what changes is the content of each step.

1. Identify. Beyond the usual confidentiality, integrity and availability, AI adds harmful or biased outcomes, unreliable output, lack of explainability, misuse, dependence on a provider and legal exposure from the data used.
2. Assess. Likelihood and impact are judged for the use case, not the technology. The same model is low risk drafting internal summaries and high risk deciding who receives credit.
3. Treat. Mitigate, transfer, avoid or accept. Acceptance is legitimate when it is documented, owned and within appetite.
4. Monitor. Because performance degrades as data shifts, reassessment on a schedule and on trigger events is itself a control.

From appetite to thresholds

Risk appetite stated as “low tolerance for unfair outcomes” cannot be audited. It becomes testable when translated into tolerances and thresholds, expressed as figures: a maximum error rate for the decision type, a limit on outcome difference between groups, a minimum share of outputs supported by source material, a response time for incidents. The auditor checks three links: appetite approved by the right level, thresholds derived from it, and monitoring that reports against the thresholds. A missing link is the finding.

Impact assessments

An AI impact assessment is done before deployment and asks who is affected, how, with what severity, and what safeguards exist. It is distinct from a general risk assessment because it takes the perspective of the people affected, not of the organisation alone. Good ones record the intended use, foreseeable misuse, affected groups, the role of human review, and a decision on whether to proceed. ISO/IEC 42001 includes an AI system impact assessment among its requirements, the NIST AI Risk Management Framework asks its Map function to characterise impacts to individuals, groups and society, and the EU AI Act attaches heavier duties to higher-risk uses; lesson 4 returns to how they differ.

The reach of an assessment is wider than the model. Look at system interactions (a model’s output may feed other systems, so one error can spread or compound), humans (people who are decided about, people who must act on the output, and staff whose work changes) and the environment (energy and resource use of training and running large models, where the organisation tracks it). An assessment that stops at the model’s accuracy has not assessed impact.

Privacy and data governance for AI

Data governance defines ownership, classification, quality standards, permitted uses and lifecycle rules for data. Privacy adds legal and ethical limits on personal data. AI strains both because training can reuse data far from the purpose for which it was collected, and because trained models can retain what they learned.

Applicable laws depend on jurisdiction, and this course gives no legal advice; the audit approach is to test controls against the obligations management has identified.

Worked example: a call-centre summariser

The insurer from lesson 1 pilots a generative tool that summarises recorded customer calls. The business owner records the risk as “low, internal use only”. The auditor’s reasoning runs as follows.

• Calls contain health details and bank information, so classification is higher than the owner assumed.
• The tool is hosted by a provider. The contract does not say whether call text is retained or used to improve the provider’s models. This is unresolved, so a purpose and retention question is open.
• No impact assessment exists. Summaries feed claims decisions, so a wrong summary can affect a customer.
• No threshold exists for summary accuracy, so the “low” rating cannot be tested.

The recommendation is not to ban the tool. It is to classify the data, obtain written terms on retention and reuse, complete an impact assessment, set an accuracy threshold with sampled review, and have the business owner accept the residual risk formally.

Easy to confuse

• Risk assessment versus impact assessment. The first weighs organisational risk; the second looks at effects on people.
• Deleting a record versus removing its influence. Dropping a row from a dataset does not remove what a trained model learned from it.
• Accepting versus ignoring a risk. Acceptance is a decision with an owner and a review date.

Key points

• AI risk assessment is use-case specific; the same model can be low or high risk.
• Appetite is only auditable once turned into thresholds that are monitored.
• Reassessment on schedule and on trigger events is an expected control because risk can shift without change.
• Impact assessments take the perspective of affected people and precede deployment.
• Privacy gaps cluster around purpose, retention, rights requests and provider reuse of data.

Check yourself

The exam-focused summary is on the governance and risk recap.