AAIR lesson 6: AI testing and operation

A model is ready for use only when testing against criteria set in advance shows that it behaves acceptably, and it stays fit for use only while someone watches how it behaves. That is the substance of two AAIR subtopics: 2B, model training, testing and validation, and 2C, implementation, maintenance and decommissioning. Three facts organise the lesson. A model learns its behaviour from training data, so training is a risk event. A single test result proves little unless the acceptance threshold was fixed beforehand. And an AI system can change its risk profile without any code change, through drift.

Training: where behaviour is created

Training is the step in which a model’s behaviour is formed from data. From a risk view the questions are about control rather than mathematics:

• Who may start a training run, with what data, and who approves it? An unapproved retraining run is a change to production behaviour.
• Is the training data suitable? Gaps, errors, bias and poisoned records all end up in the model (lesson 7 covers data quality).
• Is the process reproducible? Recording code version, data version, settings and results lets the organisation explain, repeat or roll back a result.
• Is there leakage? If test data was visible during training, performance looks better than it will be.

Testing and validation: different questions

Testing and validation are often used loosely together, but the kinds of evidence differ and are worth studying separately. A useful way to keep them apart is to ask what each answers.

A model can pass one row and fail another, so the test plan should match the risks identified at design. Independent validation relates to the three lines model: the team that built a system should not be the only one to judge it. Whatever the result, the decision rests on acceptance criteria defined before testing. Without them a test yields a number, not a decision. See the qualitative vs quantitative page for how measured results are turned into ratings.

Implementation and maintenance

Implementation moves the system into real use. Sensible controls are a documented go-live approval, a staged rollout to a limited group first, training for the people who will use it, and a fallback process if behaviour proves unacceptable.

Maintenance is where AI differs most from conventional systems. Behaviour can change in three ways:

1. The organisation retrains or tunes the model.
2. A provider updates a hosted model, which may happen without the customer’s own change record.
3. Drift. The world moves away from the training data, so accuracy or fairness degrades on a system nobody touched. Data drift is a change in the inputs; concept drift is a change in the relationship between inputs and the right answer.

Therefore AI risk belongs in change management: retraining, prompt or configuration changes, provider version changes and new data sources should each trigger a defined review. Monitoring, covered again in lesson 10, supplies the evidence that drift is occurring.

Worked example: a demand forecast at a 200-person retailer

A 200-person retailer deploys a model forecasting weekly stock needs. Before launch, the team tested it on last year’s sales and the result met the written threshold. Eighteen months later a competitor opens nearby and buying habits shift. Forecast errors grow, shelves empty, and nobody notices for weeks because no one was assigned to review accuracy.

An analyst reviewing the case identifies the gaps. No monitoring threshold existed after go-live, so nothing alerted anyone. No owner was named for model performance. Retraining happened ad hoc without a change record. The fix: assign a model owner, track forecast error weekly against a limit, schedule a periodic revalidation, and require a change ticket for any retraining. The model itself was never faulty at launch; the process around it was incomplete.

Decommissioning

Retiring a system does not end its risk. A decommissioning plan should address:

• The model artefact, which encodes what it learned and may be sensitive.
• Training and operational data, with retention and deletion duties.
• Records of past decisions, which may need to stay explainable and defensible.
• Dependencies, such as downstream processes, integrations and access accounts.
• The register entry, updated to show retired status.

Removing a person’s record from a dataset does not remove its influence from a model already trained on it, which is a reason to plan data handling before training begins.

Easy to confuse

• Testing vs validation. Testing produces evidence; validation judges whether the evidence supports the intended use, ideally independently.
• Drift vs a defect. A system can degrade with no bug and no change.
• Retraining vs routine maintenance. Retraining changes behaviour and needs change control.
• Switching off vs decommissioning. The data, artefact and decision records remain.

Key points

• Training is a risk event: control who runs it, on what data, and record versions.
• Each kind of testing answers a different question, and acceptance criteria must precede testing.
• Independent validation supports separation between builders and reviewers.
• Drift lets risk change without any system change, so ongoing monitoring and periodic revalidation are needed.
• Change management must capture retraining, provider updates, configuration and data source changes.
• Decommissioning covers the artefact, data, decision records and dependencies.

Check yourself

This lesson covers subtopics 2B and 2C of Domain 2 (AI Life Cycle Risk Management, 21%). Review the life cycle risk recap when finished.