The honest estimate

GridSearchCV.best_score_ is the cross-validated score of the combination that scored best on those same folds. You picked the maximum of a set of noisy numbers, and the maximum of noisy numbers is biased upwards. That's selection optimism.

It's the same mechanism as lesson 3's golden rule, applied one level up: try enough candidates and one will look good by luck alone.

Nested cross-validation measures it

inner = GridSearchCV(pipe, grid, cv=3, scoring="roc_auc")
nested = cross_val_score(inner, X, y, cv=5, scoring="roc_auc")

Put the whole search inside a cross-validation loop. The inner loop picks hyperparameters; the outer loop scores the entire procedure on data the inner loop never touched. The gap between best_score_ and the nested score is the optimism.

What ours actually says

grid size        best_score_    nested      optimism
 4 candidates        0.8786      0.8789      −0.0003
40 candidates        0.8800      0.8797      +0.0003

Essentially zero. Even at ten times the grid size.

That is not what most treatments of this topic imply, and it's worth saying plainly: on this problem, nested CV would have told us nothing we didn't already know. Understanding why is more useful than performing the ritual — because it tells you when you genuinely do need it.

What optimism scales with

It grows with:

• How many genuinely different things you tried. Four similar values of C are nearly the same model; forest vs SVM vs boosting vs linear are not.
• How noisy each estimate is — which means small data and high-variance models.
• How many times you have been round the loop. Twenty rounds of "let me try one more thing" is a search whose size you never counted.

And it shrinks with data. Ours has 900 rows and 180 per fold, so each fold's estimate is reasonably stable, and picking the best of eight stable numbers barely inflates anything.

Flip those around and it bites hard: 80 rows, sixteen per fold, forty candidate model families. There, best_score_ can be several points above the truth.

The cost

plain search    8 combos × 5 folds              =  40 fits
nested          (8 combos × 3 inner) × 5 outer  = 120 fits

And note what nested CV does not give you: a model. Each outer fold picks its own hyperparameters and they may differ from each other. It estimates how well the procedure generalises. The model you ship comes from one final search on all the training data.

What to actually do

1. Hold out a test set, before anything else.
2. Search on the training data with cross-validation.
3. Report the test score — never best_score_.
4. Reach for nested CV when you have little data, many genuinely different candidates, or no test set to spare.

The held-out test set was already doing this job. That's why lesson 3 made such a point of keeping it sealed — it's your defence against every kind of selection optimism at once, and it costs nothing extra.

The failure this really guards against

Not the grid. The human loop. You try a model, check, adjust, check, add a feature, check. None of that appears in any cv_results_, and it is a search — usually a much bigger one than the grid you wrote down.

No amount of nesting protects you from it. Only an untouched test set does, which is the whole argument for the discipline.