MLflow basics: tracking, models, registry

MLflow is an open-source framework that records how a machine learning model was built (tracking), packages it in a standard format (models) and manages its versions (registry). The decision rule for exams: if a scenario says compare training attempts, reproduce a result, or know which version is deployed, the answer is MLflow: tracking for experiments, the registry for lifecycle. The same MLflow API is used on Databricks, Amazon SageMaker AI and Azure Machine Learning, which is why it appears in exams from all three vendors.

The objects, in one table

Nested (child) runs group related runs under one parent, for example one child per cross-validation fold or per hyperparameter trial. Autologging (mlflow.autolog()) records parameters, metrics and the model for supported libraries such as scikit-learn, XGBoost, PyTorch and Keras without explicit log calls.

Tracking: what a training script records

import mlflow
from mlflow.models import infer_signature
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error

mlflow.set_experiment("demand-forecast")

with mlflow.start_run(run_name="rf-depth-8") as run:
    model = RandomForestRegressor(max_depth=8).fit(X_train, y_train)
    rmse = mean_squared_error(y_test, model.predict(X_test)) ** 0.5

    mlflow.log_param("max_depth", 8)
    mlflow.log_metric("rmse", rmse)
    mlflow.sklearn.log_model(
        model,
        artifact_path="model",
        signature=infer_signature(X_train, model.predict(X_train)),
    )

Newer MLflow 3 releases name the model argument name; check the version your platform runs. By default runs are written to a local mlruns directory; a tracking server or a managed platform stores them centrally instead.

A logged model is a directory with an MLmodel YAML file listing its flavors. The python_function (pyfunc) flavor is the generic interface: any model logged with a built-in flavor can be loaded with mlflow.pyfunc.load_model() and called through one predict method, which is what lets serving tools host models from different libraries. The signature describes the input and output schema, and an input example is a sample of valid input. If you log an example without a signature, MLflow infers one.

From run to registered model

Registering turns a logged model into a managed asset. On Databricks the hosted registry lives in Unity Catalog:

from mlflow import MlflowClient

mlflow.set_registry_uri("databricks-uc")   # the default in MLflow 3
name = "main.ml_team.demand_forecast"       # catalog.schema.model

mv = mlflow.register_model(f"runs:/{run.info.run_id}/model", name)
MlflowClient().set_registered_model_alias(name, "champion", mv.version)

prod = mlflow.pyfunc.load_model(f"models:/{name}@champion")

Moving the champion alias to a new version changes what every job that loads models:/<name>@champion receives, with no code change. Per Databricks documentation, Unity Catalog does not support the old stages (Staging, Production); environments are expressed through the catalog or schema and promotion through aliases. Models in Unity Catalog need a signature, use a three-level name, and need CREATE MODEL on the schema. Benefits over the workspace registry include central access control, auditing, lineage and access from every workspace attached to the same metastore, which ties into Unity Catalog.

Finding the best run

best = mlflow.search_runs(
    experiment_names=["demand-forecast"],
    order_by=["metrics.rmse ASC"],
    max_results=1,
)
best_run_id = best.iloc[0]["run_id"]

search_runs returns a pandas DataFrame, one row per run. Without order_by, results are sorted by start time, newest first. MlflowClient offers the same search as a client method, which is the form some Databricks objectives name (“identify the best run using the MLflow Client API”).

The same API on three platforms

Which exams test this

How exam questions are built around MLflow

• Tracking versus registry. “Which settings produced the best model” is tracking (params, metrics, runs). “Which version serves production and how do we roll back” is the registry. Candidates mix them up.
• Aliases versus stages. In Unity Catalog the answer is an alias; stages belong to the older workspace registry.
• Artifact versus model. Any file is an artifact; a model in the MLflow format has flavors and a signature and can be served without custom code.
• Governance words (who may use the model, lineage, cross-workspace access) point to the Unity Catalog registry rather than the workspace registry.
• Platform words. “IAM policy for the tracking server” is SageMaker AI, “workspace as tracking server” is Azure ML.

For the surrounding practice see MLOps basics, LLM evaluation for MLflow tracing and scoring, and machine learning basics for the metrics you log.

Practice questions

Go deeper

• MLflow Tracking and MLflow Model Registry (MLflow documentation)
• Models in Unity Catalog (Databricks)
• Managed MLflow on Amazon SageMaker AI and MLflow and Azure Machine Learning
• Related: Delta Lake explained for the data side of reproducibility, the Databricks ML Associate study plan, and how to study for certification exams.