Overfitting and underfitting

A model can fail in two opposite ways. It can be too simple to represent the pattern, or too flexible and end up fitting the noise. From a disappointing test score they look the same — and the cures are opposite, so guessing wrong actively makes things worse.

The two failures

Underfitting — high bias. The model can't express the relationship no matter how much data you give it. A depth-1 tree asks one question:

max_depth=1     train 0.7778     test 0.7778

Both scores are bad and identical, and both sit exactly on the dummy's number. It has learned the majority class and nothing else. No gap is the signature.

Overfitting — high variance. The model has enough freedom to memorise, so it fits patterns in your specific rows that don't exist in general:

max_depth=None  train 1.0000     test 0.7333

A perfect training score and the worst test score in the whole sweep. This is lesson 1's lookup table, arrived at honestly. A big gap is the signature.

The whole sweep

depth   train    test    5-fold AUC
    1   0.7778  0.7778     0.6379
    2   0.7793  0.7822     0.7479
    3   0.8119  0.8044     0.7853   <- peak
    5   0.8459  0.7867     0.7592
    8   0.8993  0.7333     0.7113
   12   0.9778  0.7600     0.6753
 None   1.0000  0.7333     0.6279

Two things to notice, and the second is the important one.

The training score only ever goes up. It always will — more capacity always lets a model fit the rows it can see. So the training score can never tell you when to stop, which is why "our model is 98% accurate on the training data" is not a statement about anything.

The cross-validated score rises, peaks, and falls. Every model family has this shape, and finding the peak is what tuning means. Here it's depth 3.

The gap is the diagnostic

That last row is worth internalising. A test score above the training score is almost never good news, and the instinct to celebrate it has cost people a lot of time.

Every model has the dial

• Trees — max_depth, min_samples_leaf, max_leaf_nodes
• Linear models — C for logistic regression, alpha for ridge and lasso (lesson 15)
• kNN — n_neighbors, where small k is the flexible end
• Polynomials — the degree (lesson 16)
• Forests and boosting — number and depth of trees (lesson 21)

Note that "more capacity" points in different directions for different models. For kNN,n_neighbors=1 is maximum flexibility and memorises; n_neighbors=500 is maximum rigidity. Reading the direction wrong is a very common way to make a model worse while trying to fix it.

Validation curves

from sklearn.model_selection import validation_curve

train_s, val_s = validation_curve(
    model, X, y,
    param_name="decisiontreeclassifier__max_depth",
    param_range=[1, 2, 3, 5, 8, 12, None],
    cv=5, scoring="roc_auc")

That sweep, in one call. Pick the value where validation peaks — not where the gap is smallest, and certainly not where training is best.

Learning curves — would more data help?

from sklearn.model_selection import learning_curve

sizes, train_s, val_s = learning_curve(
    model, X, y, cv=5, scoring="roc_auc",
    train_sizes=np.linspace(0.1, 1.0, 8))

This one varies the amount of data rather than the model, and it answers a question you have an actual budget for:

• Validation still climbing at the right edge? More data will help. Go and collect it.
• Flat? More rows of the same kind are wasted money. You need better features or a different model.

It is the rare diagnostic that speaks directly to a spending decision, and it's worth plotting before anyone commits to a labelling effort.

So which are we?

train roc_auc 0.8171
test  roc_auc 0.8142
gap           0.0029

Our model is not overfitting at all. Which is worth sitting with, because the instinct on hearing "it's only 79.6%" is to simplify or regularise — and both would make it worse.

A gap that small means the model has extracted essentially everything these five columns contain. It's at its ceiling. More regularisation won't help; more rows of the same data won't either.

What's left is better features — and we have been throwing away three columns since lesson 4. area, rain and rider are sitting right there, unused because they're text. That's module 3.