Databricks ML Professional: model development
Model Development is worth 44% of the Databricks Machine Learning Professional exam — around 26 of the 59 scored questions, tied with ML Ops as the largest section.
It is not the same material as the ML Associate’s development section. That one asks whether you can train a model. This asks whether you can do it rigorously: validated properly, evaluated honestly, and reproducible months later.
Validation strategy
The foundation, and where professional-level questions concentrate.
- Holdout validation for large datasets where a single split is representative.
- Cross-validation when data is limited and a single split would be noisy.
- Time-based splits for temporal data. Random splitting time series lets the model train on the future and test on the past, which inflates results and fails in production.
Match the strategy to the data. A question describing time-ordered data and a random split is describing a defect.
Data leakage
The most heavily tested single concept in the section, because it is the most consequential mistake in practice.
Leakage occurs when information unavailable at prediction time reaches the training data:
- A feature derived from the outcome, or from a field only populated afterwards.
- A random split on time-series data.
- Preprocessing — scaling, imputation — fitted on the full dataset before splitting, so the training set has seen the test set’s statistics.
- Duplicate records spanning the split.
The signature is unmistakable: excellent validation performance, poor production performance, immediately rather than gradually. Any scenario with that shape is a leakage question. Gradual decline is drift, which belongs to ML Ops.
Hyperparameter tuning
- Search strategies and their cost.
- Tuning against a validation set, never the test set. Repeatedly evaluating on the test set and selecting the best result leaks it into model selection, and the test score stops being an honest estimate.
- Recognising diminishing returns — when further tuning costs more than the improvement is worth.
- Tracking every trial so the winning configuration is recoverable.
Evaluation beyond one number
Professional-level evaluation means not stopping at a headline metric:
| Consideration | Why it matters |
|---|---|
| Class imbalance | Accuracy is meaningless when positives are rare |
| Cost of error | Recall when false negatives are expensive; precision when false positives are |
| Segment analysis | A model can perform well overall and badly for one group |
| Calibration | Whether predicted probabilities mean what they claim |
| Baseline comparison | Is this better than the trivial approach, or than the incumbent? |
The instinct the exam rewards: a single aggregate number never tells you a model is good.
Reproducibility
To rebuild a model you need four things recorded: the data version, the code version, the parameters, and the environment. MLflow tracking is how that is done on Databricks.
A model that cannot be reproduced cannot be audited, explained or safely rebuilt — and at professional level, that is treated as a defect rather than an inconvenience.
Sample questions
Question 1. A team scales features across the entire dataset before splitting into training and test sets. What problem does this introduce?
- A. Data leakage — the training data is influenced by test set statistics
- B. Increased compute cost during preprocessing
- C. Underfitting, because scaling reduces signal
- D. Class imbalance in the resulting split
Show answer
Answer: A
Fitting preprocessing on the full dataset lets information from the test set influence the training data, which is leakage and inflates the validation estimate. The order does not primarily affect compute cost, it does not cause underfitting, and it does not change class balance.
Question 2. A fraud model reports 0.97 AUC overall. Investigation shows performance is strong for one customer segment and near-random for another that makes up 15% of volume. What should the practitioner do?
- A. Deploy, since the overall AUC is strong
- B. Tune hyperparameters further to raise overall AUC
- C. Investigate and address the underperforming segment before deployment
- D. Report the aggregate metric and note the segment informally
Show answer
Answer: C
Strong aggregate performance masking poor performance for a segment is exactly what segment-level error analysis exists to surface, and the response is to investigate and address that segment. Deploying on the aggregate ignores a known weakness, more tuning on the overall metric will not fix a segment gap, and reporting only the aggregate conceals it.
Question 3. Sales forecasting data spanning three years is split randomly into 80% training and 20% test. Validation results are excellent; production results are poor. What is the most likely cause?
- A. Insufficient hyperparameter tuning
- B. A random split on time-series data leaked future information; split chronologically
- C. The training cluster was undersized
- D. Concept drift since deployment
Show answer
Answer: B
A random split on temporal data lets the model train on later periods and test on earlier ones, effectively seeing the future, which inflates validation and fails in production. Insufficient tuning and an undersized cluster would not produce strong validation results, and concept drift develops gradually rather than appearing immediately.
What to practise
Take a dataset and introduce leakage three ways: a feature derived from the outcome, preprocessing fitted before splitting, and a random split on temporal data. Watch validation scores rise each time.
Then run a segment-level error analysis on a model you already trust. Finding a segment where it performs badly is uncomfortable and exactly the professional-level habit this section is testing.