Forests and boosting

Last lesson ended on trees being unstable — change a few rows, get a different tree. That sounds fatal and turns out to be exploitable: if trees disagree because of noise rather than signal, averaging many of them cancels the noise and keeps the signal. That one idea is worth an enormous amount.

Bagging

Train each tree on a bootstrap sample — the same size as the original, drawn with replacement, so some rows appear twice and about a third are left out entirely. Then average the votes.

Each tree is individually worse than one trained on all the data. Collectively they're much better. (Those left-out rows are the "out-of-bag" set, and oob_score=True scores on them for free.)

Random forests add one more trick

from sklearn.ensemble import RandomForestClassifier

RandomForestClassifier(n_estimators=300, max_features="sqrt",
                       max_depth=6, random_state=0)

At each split, only a random subset of columns is considered. Without that, every tree would seize on distance_km first and they'd all end up similar — and correlated trees don't cancel. Forcing each split to choose from a random handful makes the trees genuinely different, which is what makes the average worth taking.

Why it works

Each tree is deep: low bias, high variance. Averaging independent estimates leaves the bias exactly where it was and divides the variance. It's lesson 16's decomposition attacked from one side only.

Which is why forests are so hard to make worse. n_estimators has no overfitting risk — more trees never hurts, it only costs time. That's unusual, and it's why a forest is the strongest thing you can get without tuning anything.

Boosting: the opposite strategy

from sklearn.ensemble import HistGradientBoostingClassifier
HistGradientBoostingClassifier(random_state=0)

# tree 1: fit y
# tree 2: fit the errors of tree 1
# tree 3: fit the errors of trees 1+2
# … hundreds of shallow trees, in sequence

Sequential, not parallel. Each tree is shallow — deliberately underfitted — and each corrects its predecessors. So the ensemble reduces bias rather than variance: the exact opposite of bagging, arriving at strength from the other direction.

HistGradientBoosting is the fast modern implementation, and it handles NaN natively — no imputer step needed at all.

And boosting's risk

Because it keeps fitting residuals, boosting will overfit if you let it. Eventually the residuals are pure noise and it fits that too. Forests essentially won't. So boosting has more knobs and needs them tuned:

learning_rate=0.1      how much each tree contributes
max_iter / n_estimators  how many
max_depth=3             how complex each one is
early_stopping=True     stop when validation stalls

Lower learning rate plus more trees is the standard trade: slower, and generally better.

The scoreboard

logistic regression        0.8786 ± 0.028
forest, depth 6            0.8650 ± 0.032
hist gradient boosting     0.8621 ± 0.028
forest, unlimited          0.8586 ± 0.031
single tree, depth 5       0.8146 ± 0.022
single tree, depth 3       0.8017 ± 0.054

Every ensemble beats every single tree — 0.86 against 0.80. That part is exactly as advertised, and it's a large gain for no thought at all.

And logistic regression beats all of them. This is the third time this dataset has said the same thing: module 4 for regression, kNN last lesson, ensembles now.

Note the fold spreads though — ±0.028 to ±0.032. The gap between the forest and logistic is about one standard deviation, so by lesson 8's standard this is suggestive rather than settled. Being honest about that is the point of reporting the spread.

What to reach for

• A linear model — always. Instant, interpretable, and a hard baseline.
• Gradient boosting — usually the strongest thing on larger, messier tabular data.
• A random forest — the best result available without tuning anything.
• A single tree — when you must explain the decision to a person.
• A neural network — when the data is images, audio or text. Not this.

On tabular data, fit a linear model and a boosted one, always. Between them they tell you whether the relationship is simple, and that's usually the most valuable thing you learn in an afternoon.

One more thing they give you

model.named_steps["clf"].feature_importances_

distance_km     0.2104
prep_min        0.1956
temp            0.1157
hour            0.0876

Free, and worth treating carefully — this is impurity-based importance, which is biased towards high-cardinality columns and says nothing about direction. Lesson 25 explains why permutation_importance is usually the honest one.