Databricks ML Professional: model development

Model Development is worth 44% of the Databricks Machine Learning Professional exam — around 26 of the 59 scored questions, tied with ML Ops as the largest section.

It is not the same material as the ML Associate’s development section. That one asks whether you can train a model. This asks whether you can do it rigorously: validated properly, evaluated honestly, and reproducible months later.

Validation strategy

The foundation, and where professional-level questions concentrate.

• Holdout validation for large datasets where a single split is representative.
• Cross-validation when data is limited and a single split would be noisy.
• Time-based splits for temporal data. Random splitting time series lets the model train on the future and test on the past, which inflates results and fails in production.

Match the strategy to the data. A question describing time-ordered data and a random split is describing a defect.

Data leakage

The most heavily tested single concept in the section, because it is the most consequential mistake in practice.

Leakage occurs when information unavailable at prediction time reaches the training data:

• A feature derived from the outcome, or from a field only populated afterwards.
• A random split on time-series data.
• Preprocessing — scaling, imputation — fitted on the full dataset before splitting, so the training set has seen the test set’s statistics.
• Duplicate records spanning the split.

The signature is unmistakable: excellent validation performance, poor production performance, immediately rather than gradually. Any scenario with that shape is a leakage question. Gradual decline is drift, which belongs to ML Ops.

Hyperparameter tuning

• Search strategies and their cost.
• Tuning against a validation set, never the test set. Repeatedly evaluating on the test set and selecting the best result leaks it into model selection, and the test score stops being an honest estimate.
• Recognising diminishing returns — when further tuning costs more than the improvement is worth.
• Tracking every trial so the winning configuration is recoverable.

Evaluation beyond one number

Professional-level evaluation means not stopping at a headline metric:

The instinct the exam rewards: a single aggregate number never tells you a model is good.

Reproducibility

To rebuild a model you need four things recorded: the data version, the code version, the parameters, and the environment. MLflow tracking is how that is done on Databricks.

A model that cannot be reproduced cannot be audited, explained or safely rebuilt — and at professional level, that is treated as a defect rather than an inconvenience.

Advanced MLflow: nested runs and custom PyFunc models

The outline’s Advanced MLflow Usage block asks you to use nested runs for complex experiments, log custom metrics, parameters and artifacts in code, and build custom model objects. The Model Deployment section adds registering a custom PyFunc model with its artifacts in Unity Catalog. Both go beyond the basic tracking calls described in the MLflow basics guide.

Nested runs put related runs under one parent, so a tuning sweep or a cross-validation shows as one entry in the experiment that you can expand. Start the parent run, then start each child with nested=True:

with mlflow.start_run(run_name="tuning") as parent:
    scores = {}
    for depth in [4, 6, 8]:
        with mlflow.start_run(run_name=f"depth_{depth}", nested=True):
            scores[depth] = train_and_score(depth)
            mlflow.log_param("max_depth", depth)
            mlflow.log_metric("val_auc", scores[depth])
    mlflow.log_param("best_depth", max(scores, key=scores.get))

Each child gets its own parameters and metrics; the parent holds the summary, such as the winning configuration. Without nested=True, starting a second run while one is active raises an error rather than creating a child. A run can also be attached to an existing parent with parent_run_id.

A custom PyFunc model is what you build when the logic you want to serve is more than one library’s predict: preprocessing, feature engineering at request time, several models combined, or a lookup file. Subclass mlflow.pyfunc.PythonModel, load files in load_context(context) and do the work in predict(context, model_input). Log it with mlflow.pyfunc.log_model, passing python_model=, an artifacts dictionary (the files appear at context.artifacts[...] when the model loads), and registered_model_name="catalog.schema.model" to register it in Unity Catalog. Unity Catalog requires a model signature on every new version; pass signature or an input_example from which MLflow infers one. The result serves like any other model: deployed through Model Serving and queried by REST or the MLflow Deployments SDK, as covered in model deployment.

Pitfall: a model that works in the notebook fails at registration or serving because preprocessing lived in notebook code and a scaler file sat on the driver’s disk. The fix is to move that logic into the PyFunc class and log the file as an artifact, so the registered version carries everything it needs.

Sample questions

What to practise

Take a dataset and introduce leakage three ways: a feature derived from the outcome, preprocessing fitted before splitting, and a random split on temporal data. Watch validation scores rise each time.

Then run a segment-level error analysis on a model you already trust. Finding a segment where it performs badly is uncomfortable and exactly the professional-level habit this section is testing.