Databricks ML Associate model development
Model Development is worth 31% of the Databricks Machine Learning Associate exam — around 15 of the 48 scored questions and the second largest section. This is the modelling work itself, framed as things you do on Databricks in Python.
Preparing data for modelling
- Exploratory analysis: distributions, missing values, outliers, class balance.
- Handling missing data, and recognising that each choice is a modelling decision.
- Encoding categorical variables; scaling numeric ones where the algorithm needs it.
- Splitting into training, validation and test sets appropriately for the data — and chronologically for time series, never randomly.
Feature engineering
- Creating features that carry signal.
- Reusing feature definitions between training and inference so they are computed identically. Divergence here is training/serving skew, a classic production failure.
Training
- Common algorithm families and the problems they suit: linear models, tree ensembles, gradient boosting.
- Training in Python on Databricks, and when distributed training is warranted.
- Managing compute cost during training.
Hyperparameter tuning
- Hyperparameters are chosen before training — learning rate, tree depth, regularisation, number of estimators. Distinguish them from parameters, which the model learns.
- Search strategies, and automated tuning at scale.
- Tuning against a validation set, never the test set.
Evaluation metrics
The most reliably tested area in the section, because metric choice is where people go wrong.
| Metric | Use when |
|---|---|
| Accuracy | Classes are reasonably balanced |
| Precision | False positives are costly |
| Recall | False negatives are costly |
| F1 | You need both balanced |
| AUC-ROC | Comparing classifiers across thresholds |
| RMSE / MAE | Regression |
| R² | Proportion of variance explained |
The recurring scenario is class imbalance: a rare positive class where accuracy looks excellent because the model predicts the majority every time. Fraud, churn, defects, disease — always a precision and recall question.
Choose by cost of error. Missing a fraudulent transaction is a false negative, so recall matters. Wrongly flagging a good customer is a false positive, so precision matters.
Overfitting and underfitting
Know both by signature:
- Overfitting — strong on training data, weak on validation and test. The model memorised. Mitigations: more data, regularisation, simpler model, early stopping.
- Underfitting — weak on both. The model never learned the pattern. Mitigations: more capacity, better features, longer training.
Sample questions
Question 1. A model must flag fraudulent transactions. Missing fraud costs far more than investigating a legitimate transaction unnecessarily. Which metric should be prioritised?
- A. Accuracy
- B. Recall on the fraud class
- C. Precision on the fraud class
- D. R²
Show answer
Answer: B
Missing fraud is a false negative, so recall on the fraud class is the metric to prioritise, accepting more false positives as the cheaper error. Accuracy is uninformative at this class balance, precision minimises the less costly error here, and R squared applies to regression.
Question 2. A model achieves 0.97 on training data and 0.61 on the validation set. What is happening and what is an appropriate response?
- A. Underfitting; train for more epochs
- B. Normal behaviour; register the model
- C. Overfitting; apply regularisation, simplify the model, or obtain more data
- D. A compute problem; use a larger cluster
Show answer
Answer: C
A large gap between strong training performance and weak validation performance is overfitting, addressed by regularisation, a simpler model or more data. Training longer worsens it, a larger cluster changes compute not fit, and switching the metric hides the problem rather than fixing it.
Question 3. A data scientist tunes hyperparameters by repeatedly evaluating against the test set and selecting the best result. What is wrong with this?
- A. The test set is no longer an unbiased estimate of unseen performance; tune on a validation set
- B. It is correct but computationally slow
- C. The wrong metric is being optimised
- D. Too many hyperparameters are being tuned at once
Show answer
Answer: A
Tuning against the test set leaks information from it into model selection, so the test score no longer estimates performance on unseen data; a separate validation set should be used for tuning. The approach is not simply slow, the metric is not inherently wrong, and the issue is not the number of hyperparameters.
What to practise
Train one model on a deliberately imbalanced dataset and print accuracy, precision, recall and F1 side by side. Then overfit a model on purpose — a deep tree on a small dataset — and compare training and validation scores.
Those two exercises produce the exact numeric patterns the exam describes in words, and seeing them once makes the questions immediate.