AAIA lesson 8: Testing AI solutions

No single test shows that an AI solution is fit for use. Accuracy answers whether outputs are usually right, robustness whether they stay right under odd input, fairness whether errors fall evenly across groups, and red teaming whether a motivated adversary can break it. Subtopic 2E, Testing Techniques for AI Solutions, expects an auditor to know which test answers which question and to ask for dated, version-tagged results rather than a statement that “testing was done”.

Verification and validation

Verification asks whether the system was built to specification: requirements traced to design, code reviewed, pipeline run as documented. Validation asks whether the system is fit for its intended purpose in its real context: does the model perform acceptably on data it has never seen, for the people it will affect? A model can pass verification and fail validation if the specification itself was wrong. Auditors care most about validation, and about whether it was performed by someone independent of the builders. Validation also asks whether the model’s logic and outputs serve the stated business objectives and policies, not only whether a metric is met.

Metrics at concept level

For a model that flags something (fraud, a risky applicant, a harmful message), four outcomes exist: true positives, false positives, true negatives and false negatives.

• Precision: of the cases flagged, how many were truly positive. Low precision means many false alarms.
• Recall: of the truly positive cases, how many were caught. Low recall means missed cases.
• Accuracy: the share of all calls that were right. It misleads when positives are rare: a model that never flags fraud can be 99% accurate if only 1% of cases are fraud.

Which error is worse is a business and ethical judgement, not a technical one. Missing a fraud costs money; wrongly denying a benefit harms a person. The auditor checks that the acceptance threshold was set in advance, by a named owner, with that trade-off in view. For generative systems, accuracy gives way to measures such as groundedness against source documents and rates of unsafe output, assessed against a fixed evaluation set; see LLM evaluation and ML basics: overfitting and metrics.

A further trap is overfitting: a model memorises training data, including its noise, and looks excellent there but worse on new cases. Lesson 5 separated the sets: the validation set guides tuning during development, and the test set is held back for one final estimate. Results are credible only when measured on a test set that was never used for tuning.

Test types and what each shows

Worked example

A lender introduces a model that pre-screens loan applications. Management supplies a deck stating 94% accuracy. An auditor would ask what share of applications are defaults (if it is 5%, 94% proves little), then request precision and recall at the chosen cut-off, the date and dataset of the test, and the approval of the threshold. Next comes group-level results: approval rates and false-negative rates across age bands and other relevant groups, because an average can hide a group the model serves poorly. Then robustness (what happens with missing income data) and the date of the last regression run.

Finally comes independent re-performance. The auditor cannot recompute the model’s internals, but can ask for the test dataset and the model version, run it through the model in a controlled setting, and compare results with those reported. Differences prompt questions about version control and data leakage between training and test sets.

Easy to confuse

• Precision vs recall: precision penalises false alarms, recall penalises misses.
• Fairness testing vs bias review of data: one measures outcomes, the other inspects inputs; both are needed.
• Selection-rate parity vs equal error rates: the first compares how often each group receives the favourable outcome, the second compares false positives and false negatives by group. When groups differ in underlying rates, the two generally cannot both be met, so the auditor checks that the chosen measure was reasoned and documented.
• Red teaming vs penetration testing: the focus differs and the scopes overlap. AI red teaming tries to provoke harmful, leaked or unintended model behaviour, such as jailbreaks; penetration testing looks for exploitable weaknesses in infrastructure, applications and access.
• Explainability vs transparency: explainability concerns why an output arose; transparency concerns disclosure that AI is used and how.

Key points

• Validation (fit for purpose) matters more to the auditor than verification (built to spec), and should be independent.
• Accuracy alone misleads when outcomes are imbalanced; ask for precision, recall and the threshold rationale.
• Fairness needs group-level results; averages hide disparities.
• Evidence should be dated, version-tagged and tied to a dataset.
• Re-performance on a controlled sample and a fixed benchmark check the reported results.

Check yourself

The domain recap is on the AI operations page.