AAIA lesson 11: Sampling and evidence

Sampling an AI system means sampling its outputs and decisions, not its code, and the evidence that matters is records of tests, monitoring and review rather than inspection of logic. A usable approach defines the population precisely (which model version, which period, which decision type), picks statistical or judgemental selection on purpose, and ranks evidence by how independent and direct it is. These are subtopics 3B (Audit Testing and Sampling Methodologies) and 3C (Audit Evidence Collection Techniques).

Defining the population

Traditional audit populations are transactions: invoices, access requests, changes. For AI the population might be all decisions made by version 4.2 of a model between two dates, all prompts handled by a chatbot in a month, or all retraining runs in a year. Three decisions come before any selection:

• Unit: a decision, a conversation, a document summary, a retraining event.
• Completeness: is the log complete? A sample from a population missing a day of records is not a sample of the population. Reconcile counts with an independent source such as case-management volumes.
• Stratification: split by version, channel, risk level or group, because averages across versions blur results.

Statistical and non-statistical sampling

Four terms carry the method (sampling risk differs from non-sampling risk, which comes from a badly designed or misread test and exists even when every item is tested). Sampling risk is the chance that the sample does not represent the population, so the conclusion would differ if every item were tested. Tolerable error is the most error the auditor will accept and still conclude the objective is met; in a control test it is the highest acceptable rate of deviation from the control. The confidence level is how sure the auditor wants to be. Raising confidence, lowering tolerable error or expecting more errors all increase the sample size. Attribute sampling records whether a characteristic is present or absent (the reviewer overrode the recommendation or did not, the output was correct or not) and yields a rate, which suits control tests on AI decisions. Variable sampling estimates an amount, such as the value of claims wrongly paid, and suits substantive tests.

A sensible AI audit mixes both selection styles: a random sample to estimate rates, plus a targeted set of risky cases reported separately. Where decisions number in the millions, analytics over the whole population (lesson 12) may replace sampling for some tests. For comparing versions, a fixed benchmark set holds inputs constant; random samples from different periods do not.

Evidence types and reliability

Audit techniques include inquiry, observation, inspection and re-performance. Two separate tests apply to evidence. It is sufficient when there is enough of it to be convincing, so that another auditor would reach the same conclusion (a question of quantity). It is reliable when it is valid, factual, objective and supportable (a question of quality), and also relevant to the objective. Reliability rises when evidence is obtained directly by the auditor, comes from independent sources, exists in documentary or system-generated form and is dated. Typical ordering, strongest first: re-performance or automated testing across the period, inspection of system-generated records, observation, inquiry, management representation alone. Re-performance of a model’s output is possible by running recorded inputs through it, allowing for run-to-run variation; what cannot be re-performed is its internal reasoning.

For AI, the useful artefacts are:

• Validation and test reports tied to a version, dataset and date.
• Model, prompt and data version logs and change approvals.
• Monitoring output with thresholds and responses to breaches.
• Review and override logs.
• Data lineage records and source agreements.

Evidence the auditor generates, such as running prompts through the system, carries more weight than reports prepared by the team being audited. AI also makes fabricated evidence easy: a document, screenshot or log can look flawless and still be synthetic. Obtain records directly from the source system and check authenticity before relying on anything that arrived from the auditee.

Vendors and black-box models

Many organisations use models they cannot open. The auditor cannot inspect weights or training data, so assurance comes from layers:

1. Contract: audit and notice rights, data use terms, change notification, incident reporting, location of processing.
2. Independent assurance: third-party assurance reports, certifications such as an ISO/IEC 42001 certification where held, and model documentation from the provider. These show the provider’s controls, within the certificate’s stated scope, not fitness for the buyer’s use.
3. Own testing: black-box testing through the interface with the organisation’s inputs, using a fixed benchmark, fairness checks and adversarial prompts.
4. Monitoring by the user: logging inputs and outputs, tracking quality, and checking after any provider change.

The conclusion should say what could and could not be verified. A finding that “the provider’s training data cannot be examined” is a limitation to disclose, with compensating controls assessed, not necessarily a control failure.

Worked example

The insurer uses a vendor’s text classifier to route complaints. The auditor wants assurance that regulatory complaints are routed correctly. Population: all complaints routed by the current version between January and June, reconciled to the complaints register (a 2% count mismatch explained). Stratified by channel. A random sample of 150, sized from the chosen confidence level and the tolerable error rate, is re-routed by the auditor’s own reading; a targeted set adds 30 cases the model flagged as low-confidence plus 20 from customers who later escalated. The auditor treats the vendor’s accuracy figure as background only, requests the vendor’s latest assurance report, and runs a fixed set of 100 test complaints before and after the vendor’s mid-year update.

Easy to confuse

• Vendor documentation vs evidence about this deployment: the first describes the product, the second shows it working in the organisation’s context.
• Statistical sample vs risk-based selection: only the first allows projection.
• Sufficient vs reliable evidence: sufficient is how much; reliable is how good. A large pile of auditee-prepared summaries can be sufficient in volume and still unreliable.

Key points

• Define unit, period, version and completeness before sampling AI outputs.
• Compare observed error or deviation rates with tolerable error; a single exception is judged on its rate and nature, not automatically a control failure.
• Combine random samples for rates with judgemental selection for risky cases.
• Evidence generated by the auditor outranks evidence supplied by the audited team.
• For black boxes, layer contract rights, independent assurance, own testing and user monitoring, and disclose limitations.

Check yourself

The domain recap is on the audit tools and techniques page.