Overfitting and ML Metrics Explained

Overfitting means a model has memorised its training data and scores far better there than on data it has not seen; underfitting means it is too simple to score well anywhere. The decision rule for the metric questions is: pick the metric by the cost of the mistake. If a missed positive is expensive, optimise recall; if a false alarm is expensive, optimise precision; if the classes are imbalanced, never trust accuracy alone.

Three datasets, three jobs

A model is judged on data it did not learn from, otherwise the score only measures memory.

Every time a decision is made by looking at the validation score, the validation set leaks a little into the model. That is why a separate test set exists. Two rules recur in exam scenarios: split before any preprocessing that learns from the data (otherwise information leaks across the split), and split time-series data chronologically, never randomly. When data is scarce, k-fold cross-validation repeats the train/validate split k times and averages the result; scikit-learn documents it in its cross-validation guide.

Reading the gap: overfitting or underfitting

Compare the training score with the validation score.

The trap is the opposite fix: adding model capacity to an overfit model makes it worse, and adding regularisation to an underfit model does the same. Plotting training and validation loss against epochs (Google’s crash course calls this a generalization curve) makes it visible: the two curves move together at first, then training loss keeps falling while validation loss turns and rises. Early stopping halts training at that turning point.

A related failure is data leakage: a feature that encodes the answer, or contains information only available after the event. It produces excellent test scores and poor production results, which looks like overfitting but is not cured by regularisation.

One confusion matrix, every metric

Worked example. A fraud model scores 1,000 transactions, of which 50 are actually fraud. It flags 60 as fraud, and 40 of those flags are correct.

The counts add up: 40 + 10 + 20 + 930 = 1,000. Now each metric:

• Accuracy = (TP + TN) / total = 970 / 1,000 = 0.97.
• Precision = TP / (TP + FP) = 40 / 60 = 0.667. Of the flags raised, two thirds were real fraud.
• Recall (sensitivity, true positive rate) = TP / (TP + FN) = 40 / 50 = 0.80. Of the real fraud, 80% was caught.
• F1 = harmonic mean of precision and recall = 2TP / (2TP + FP + FN) = 80 / 110 = 0.727.
• Specificity = TN / (TN + FP) = 930 / 950 = 0.979.

Notice the 97% accuracy. A useless model that always answers “legitimate” scores 950 / 1,000 = 95% accuracy with recall 0. Accuracy rewards the majority class, so with rare positives it hides the failure. This is the imbalance trap behind a large share of metric questions.

Threshold. Most classifiers output a probability, and a threshold turns it into a label. Raising the threshold flags fewer items: fewer false positives (precision up), more missed positives (recall down). Lowering it does the reverse. Precision and recall trade against each other, and F1 summarises one point on that trade.

AUC. The ROC curve plots true positive rate against false positive rate across all thresholds; the area under it (AUC-ROC) is 1.0 for a perfect ranker and 0.5 for random guessing. It measures how well the model ranks positives above negatives, independent of any one threshold. With very rare positives the precision-recall curve (AUC-PR) is usually more informative, because the false positive rate stays small even when there are many false alarms in absolute terms. Google’s Machine Learning Crash Course covers both.

For regression the equivalents are error measures such as mean absolute error and RMSE, where lower is better.

Which exams test this

How the questions are built

Scenario questions give a symptom and ask for the cause or the metric. The signals worth recognising:

• “Excellent on training data, poor on new data” is overfitting. “Poor on both” is underfitting.
• “Rare positive class” or “99% accuracy but nothing detected” means accuracy is the wrong metric.
• Constraint words about cost decide recall versus precision: “must not miss”, “life-threatening”, “any case” point to recall; “avoid false alarms”, “costly manual review”, “must be certain” point to precision.
• “Compare classifiers regardless of threshold” points to AUC. “Balance precision and recall” points to F1.
• “Great in testing, worse in production immediately” suggests leakage or a training/serving mismatch, while a slow decline suggests drift (see MLOps basics).

Practice questions

Go deeper

• Classification: accuracy, precision, recall, Google Machine Learning Crash Course
• Overfitting, Google Machine Learning Crash Course
• Metrics and scoring, scikit-learn documentation
• Related on this site: MLOps basics, LLM evaluation, responsible AI principles and how to study for certification exams