Cross-validation

Lesson 3 left a problem on the table. The same model across eight random seeds scored anywhere from 0.796 to 0.836 — four points of pure luck. A single train/test number is one draw from a distribution, and reporting it alone hides how wide that distribution is.

Rotate the test set

Split the data into k folds. Train k times, each time holding out a different fold. Every row gets to be test data exactly once, and you finish with k scores instead of one.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
scores          # [0.815  0.785  0.819  0.824  0.828]
scores.mean()   # 0.8142
scores.std()    # 0.0154

One detail that catches everyone: pass the unfitted estimator. cross_val_score clones it and fits a fresh copy per fold. Handing it something already fitted doesn't error — it silently refits, and if you then inspect your original object it's not the one that produced the scores.

Report the spread

0.814 ± 0.015. That's the honest form: a centre and a spread. And the spread is directly useful — it's your threshold for belief.

A model that scores 0.010 better than another, when fold-to-fold variation is 0.015, has not been shown to be better at all. Most "we improved it by 1%" claims die right here, which is exactly why the habit is worth building.

Stratify here too

For classification with an integer cv, scikit-learn uses StratifiedKFold automatically — each fold keeps the class balance. That's the sensible default, and it's worth knowing it happens.

Pass a plain KFold and you lose it. With a rare class that can produce a fold with almost no positives, or none at all, at which point precision is undefined and your mean quietly includes a zero.

from sklearn.model_selection import StratifiedKFold, KFold, GroupKFold, TimeSeriesSplit

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)   # explicit
cross_val_score(model, X, y, cv=cv, scoring="f1")

Note shuffle=True. Without it, folds are contiguous blocks of your data in its original order — which is a problem if that order means anything, and it usually does.

How many folds

 cv=3    AUC 0.811 ± 0.014     600 train / 300 test per fold
 cv=5    AUC 0.814 ± 0.015     720 train / 180 test per fold
 cv=10   AUC 0.815 ± 0.038     810 train /  90 test per fold

The estimate barely moves. What changes is how shaky each individual fold is: 10-fold tests on only 90 rows at a time, so each score is noisier even though the mean is slightly less biased.

So more folds is not more rigour — it's more compute for a slightly better estimate, paid for with noisier components. 5 is the sensible default. Go to 10 when data is scarce and every training row counts. The extreme is LeaveOneOut: n folds, each testing on one row, nearly unbiased and wildly variable.

cross_validate, for more than one number

from sklearn.model_selection import cross_validate

res = cross_validate(model, X, y, cv=5,
                     scoring=["accuracy", "precision", "recall", "f1", "roc_auc"],
                     return_train_score=True)

res["test_accuracy"].mean()    # 0.8167
res["test_recall"].mean()      # 0.3600
res["test_roc_auc"].mean()     # 0.8142
res["train_roc_auc"].mean()    # 0.8171

return_train_score=True is the flag worth remembering. The gap between train and test scores is how you diagnose overfitting, and it's the whole subject of the next lesson. Ours is 0.003 — which is not the good news it sounds like.

The scoring argument takes any name from sklearn.metrics.get_scorer_names(), or a callable you make with make_scorer. Note that scorers are always "higher is better", so error metrics appear negated — "neg_mean_absolute_error". That sign trips up everyone once.

When plain k-fold is wrong

Same list as lesson 3, because it's the same assumption:

• Time-ordered rows → TimeSeriesSplit. Each fold trains on the past and tests on the future; folds grow rather than rotate.
• Grouped rows → GroupKFold, so the same customer never appears in both halves of a fold.
• Both → StratifiedGroupKFold exists for exactly this.

What cross-validation is for

This is the point of the whole lesson, so it's worth stating plainly. Cross-validation is how you make decisions without touching the test set.

Which model, which hyperparameters, which features, which threshold — all of it happens inside the training data, measured by cross-validation. The test set is opened once, at the end, and whatever it says is what you report. That division of labour is the discipline, and module 6 automates it.

One warning to carry forward: any preprocessing must live inside the estimator you pass to cross_val_score. Scale the whole dataset first and every fold's "test" rows have already influenced the scaling — the scores come back optimistic and nothing warns you. That's what Pipeline is for, and it's lesson 13.