MLOps Basics: Registry, CI/CD, Drift
MLOps applies DevOps discipline to machine learning: every model is versioned in a registry, built and promoted by an automated pipeline, and monitored in production, because a model degrades even when no code changes. The decision rule for exam scenarios is: if quality falls while the infrastructure is healthy and nothing was deployed, the answer is drift monitoring plus a retraining path; if a change must reach production safely, the answer is a pipeline with a validation gate and a registry, not a manual copy.
Why ML needs more than ordinary CI/CD
Ordinary software behaves the same tomorrow as today unless someone changes it. A model’s behaviour comes from code, data and parameters, and the data moves. So an ML pipeline has three things to version and test, not one, and a fourth stage that ordinary software lacks: watching the model after release. Compare CI/CD explained for the generic loop; this guide covers what is added for models.
The loop, stage by stage
| Stage | What happens | What goes wrong without it |
|---|---|---|
| Track experiments | Parameters, metrics and artifacts logged per run | Nobody can reproduce the good run |
| Train in a pipeline | Data prep, training and evaluation as repeatable steps | Results depend on one person’s notebook |
| Register | The model becomes a named, versioned entry with lineage back to its run | No record of what is live or how it was built |
| Validate | Candidate is compared with the current model and checked against thresholds | A worse model is promoted automatically |
| Deploy | Real-time or batch endpoint, with staged traffic | Whole user base hit by a bad release |
| Monitor | Input, prediction and quality signals compared with a baseline | Silent degradation |
| Retrain | Scheduled or triggered, then back through validation | Stale model, or an unvetted replacement |
The model registry
A registry is the central store that lets a team manage a model’s lifecycle. MLflow’s registry, for example, provides lineage (which run produced the model), versioning, aliases and tags. Each registration creates a new version, and an alias such as @champion identifies which version serves production (MLflow deprecated the older Staging/Production stages in version 2.9). Promotion means moving that alias, rollback means moving it back, and serving code references the alias instead of a file path. Azure Machine Learning, Amazon SageMaker and Databricks Unity Catalog each have an equivalent; exams use the vendor’s own names, but the concepts are the same: version, lineage, approval status, promotion, rollback.
CI/CD for ML
Continuous integration tests code and also data and model quality: schema checks, unit tests of feature code, and an evaluation run with a pass threshold. Continuous delivery builds the model artifact, registers it and deploys it, usually behind an approval step. The safe deployment patterns match general software: blue/green or canary releases split traffic between versions, and mirrored (shadow) traffic tests a candidate on live requests without returning its answers. The constraint word to spot is “without affecting users”, which points to shadow testing or a small traffic percentage.
Monitoring and the two kinds of drift
Monitoring has three layers: infrastructure (latency, errors), pipeline (did the job run), and model (are predictions still good). Green infrastructure dashboards prove nothing about the model. Model monitoring compares production data with a reference, usually training data. Azure Machine Learning describes this as computing statistics on production inference data against reference data with thresholds that raise alerts, with built-in signals for tabular data: data drift, prediction drift, data quality, feature attribution drift and model performance. Amazon SageMaker Model Monitor offers the same idea, although AWS closed it to new customers and points them to open-source monitoring solutions with Amazon QuickSight and CloudWatch. Every approach needs reference data to compare against, usually the training data or a recent window of production data.
- Data drift: the input distribution changes (a new customer segment, a source system reformats a field).
- Concept drift: the relationship between inputs and the correct answer changes while inputs look the same (fraudsters adapt). It is only visible through outcome quality, which arrives late.
- Training/serving skew: features computed differently at training and inference. It hurts immediately after deployment, unlike drift, which is gradual.
Worked example: a churn model that decays
A churn model is registered as version 3 and serves through an alias. Eight months later, retention campaigns built on it perform worse. Endpoint latency and error rates are normal, nothing was deployed. Diagnosis steps: compare current feature distributions with the training baseline (a shifted age and plan mix means data drift); if distributions match but accuracy against confirmed churn labels fell, concept drift. Response: a scheduled or alert-triggered pipeline retrains on recent data and registers version 4; the validation step compares it with version 3 on a held-out recent set; only if it wins is the alias moved, with canary traffic first and version 3 kept for rollback. Skipping the comparison is the classic flaw: newer is not automatically better.
Which exams test this
| Exam | How the topic appears | Where on this site |
|---|---|---|
| AI-300 | Implement machine learning model lifecycle and operations (25–30%): MLflow tracking, training pipelines, registering and archiving MLflow models, real-time and batch endpoints, progressive rollout and rollback, data drift and retraining or alert triggers; Design and implement an MLOps infrastructure (15–20%): workspaces, shared registries, IaC and GitHub Actions | Model lifecycle, ML infrastructure |
| MLA-C02 | Deployment and Orchestration (24%): CI/CD pipelines, automated deployment and rollback, retraining mechanisms, versioning in SageMaker Model Registry or MLflow; Operating, Monitoring, and Securing (24%): drift detection and changes in data distribution; ML Model and FM Development (24%): performance baselines, drift detection and shadow variants | Deployment and orchestration, Monitoring |
| Databricks ML Professional | MLOps (44%): lifecycle pipelines, unit and integration tests, Asset Bundles, automated retraining on drift or degradation, and Lakehouse Monitoring for drift and alerts; Model Deployment (12%): blue-green and canary rollout | MLOps |
| Databricks ML Associate | Databricks Machine Learning (38%): MLOps best practices, MLflow tracking, registering models in the Unity Catalog registry, promoting code versus models, and champion and challenger aliases | Databricks Machine Learning |
| AIF-C01 | Fundamentals of AI and ML (20%), task 1.3: describing MLOps concepts such as experimentation, repeatable processes, production readiness, model monitoring and re-training. Concept level only; building pipelines is out of scope | AIF-C01 fundamentals of AI and ML explained |
How the questions are built
- “Accuracy declining for months, no changes, healthy endpoint” is drift. “Fine in validation, bad immediately in production” is skew or leakage, not drift.
- “Feature distributions unchanged but accuracy fell” is concept drift; “input data looks different” is data drift.
- “Automatically retrain” has a hidden half: validate against the incumbent before promotion.
- “Roll back quickly” or “know which model served a prediction” points to the registry and version logging.
- “Alert when” points to monitoring thresholds that trigger an event or pipeline, not someone watching a dashboard.
Practice questions
Question 1. A fraud model on a managed endpoint has lost accuracy gradually over six months. Latency and error metrics are normal and no deployment occurred. Fresh outcome labels are available weekly. What should the team implement?
- A. Scale the endpoint out to more instances
- B. Restart the endpoint weekly
- C. Add more CPU and memory alarms
- D. Monitor drift against the training baseline and trigger retraining
Show answer
Answer: D
Gradual decay with healthy infrastructure is drift. Monitoring against a training baseline plus a retraining pipeline addresses it. Scaling the endpoint fixes throughput, restarting fixes transient faults, and more infrastructure alarms do not measure prediction quality.
Question 2. A pipeline retrains a model nightly. The team wants to avoid ever promoting a model that is worse than the one in production. Which control should be added?
- A. An automated evaluation gate that compares the candidate with the production version before registration or promotion
- B. Retrain every hour instead of nightly
- C. Delete previous model versions to save storage
- D. Deploy each new model to all traffic immediately
Show answer
Answer: A
A validation step that compares the candidate with the current production model on the same evaluation data, and promotes only on improvement, is the control. Retraining more often does not prevent worse models. Deleting old versions removes the rollback path. Deploying to all traffic at once removes the chance to catch a regression.
Go deeper
- MLflow Model Registry, MLflow documentation
- Model monitoring in production, Microsoft Learn
- Amazon SageMaker Model Monitor, AWS documentation (closed to new customers; existing customers keep access)
- Related on this site: CI/CD explained, ML basics: overfitting and metrics, LLM evaluation, AI governance and risk