AAISM lesson 12: Trust and monitoring

Trust and safety controls turn principles such as fairness and transparency into things that can be tested and evidenced, and monitoring shows whether they keep working. Subtopics 3D and 3E of the AAISM outline cover them, and two supporting tasks are directly relevant: conduct risk-based human oversight of AI inputs and outputs, including trust and safety, quality, explainability and robustness; and define and monitor security metrics for AI solutions. Both belong to Domain 3, at 38% of the exam.

From principle to control

The principles on the responsible AI page are only useful to a manager when each has a mechanism.

Bias and fairness. Bias can enter through data, labels or deployment context. Testing compares outcomes across groups and records results against a threshold set by the business and legal functions; a single global accuracy figure can hide a large gap.

Explainability. The level needed depends on the decision. A product recommendation needs little; a declined application needs reasons a person can understand and contest.

Safety guardrails. Filters and policies applied to inputs and outputs reduce harmful content and policy violations. They are one layer and are tested through red teaming, because filters can be bypassed.

Human oversight, sized by risk

Oversight is not a human glancing at everything. It is matched to impact: low-impact outputs may be sampled after the fact, medium-impact ones reviewed on exception, and consequential decisions (denying a claim, dismissing a person, moving money) need a person with the information, time and authority to override. Oversight fails when the reviewer rubber-stamps because volume is high or the model appears confident, so the process measures override rates and reviewer workload.

Monitoring and detection

Infrastructure monitoring does not show that a model is being abused or has drifted. AI monitoring adds:

• Prompts and outputs, within privacy limits, to see abuse and leakage.
• Quality signals such as groundedness, error rates and fairness measures over time, which reveal drift.
• Query patterns such as high-volume systematic probing, which may indicate extraction.
• Safety events: what filters blocked and what got through.
• Agent actions: tool calls and the identity used, because these are the consequential events.

These feed detection rules and the incident process from lesson 5 (see incident response lifecycle). Logging must itself respect privacy: logs of prompts can hold personal data, so retention and access are limited.

Metrics and tools

Good AI security metrics are tied to a decision. Examples: share of AI systems in the inventory with a named owner and a current risk assessment; rate of successful injection in red-team tests; time to detect and contain AI incidents; proportion of vendor models with change notice in contract; drift measures against thresholds; override rate for human review. Metrics that only count activity, such as number of prompts, say little about risk.

Tools support the program rather than replace it: gateways and guardrail services, AI discovery that finds unsanctioned use, data loss prevention (see DLP explained), model scanning, evaluation platforms and the SIEM that correlates the logs. Reviewing and implementing them is itself a supporting task, assessed like any security tool for coverage, integration and who operates it.

Worked example

A 400-person insurer uses a model to pre-screen claims for investigation. Bias testing shows that the referral rate for one customer age group is twice that of others for similar claims. The risk owner sets a maximum acceptable disparity, the model is retuned, and the test becomes a monthly check. Because a referral affects a customer, the process requires an investigator to review each referral with the model’s main reasons visible; the team tracks how often investigators disagree and finds the rate dropping toward zero, which triggers a review of whether reviewers are simply agreeing. In monitoring, the security team sees one account submitting thousands of near-identical claims with small variations and flags it as possible probing.

Easy to confuse

• Explainability vs transparency. Explainability is about why a decision was made; transparency is about disclosure that AI is used and how it was built.
• Accuracy vs fairness. A model can be accurate overall and unfair to a subgroup.
• Human in the loop vs human on the loop. In the loop, a person approves each decision; on the loop, a person supervises and can intervene.

Key points

• Every principle needs a testable control and evidence.
• Fairness is measured by group, against agreed thresholds.
• Oversight is sized to impact, and its effectiveness is itself measured.
• AI monitoring covers prompts, outputs, quality, query patterns and agent actions.
• Metrics should support a decision, not just count activity.

Check yourself

This lesson covers subtopics 3D and 3E of Domain 3, AI Technologies and Controls (38%); the recap is at technologies and controls explained.