Decision trees

A decision tree is a nest of if-statements that it learned by itself. Each node asks one question about one column; each leaf is a prediction. And unusually for a model, you can read the whole thing:

if prep_min <= 8.5:
    if distance_km <= 2.4:  predict "on time"
    else:                    predict "on time"
else:
    if area == "Industrial": predict "late"
    else:                    predict "on time"

That isn't pseudocode for the model — it is the model. Which makes it the one family you can hand to somebody with no statistics and have them follow the reasoning.

How it chooses splits

At each node it tries every column at every threshold and keeps whichever makes the two sides purest. Purity is Gini by default or entropy if you ask; they almost always pick the same splits, and the difference matters far less than one step of max_depth.

This is also why trees need no scaling: each candidate split is only ever compared against others in the same column, so the units cancel out.

Greedy, and that has consequences

It picks the best split available right now and never reconsiders. So a pair of splits that are mediocre alone but excellent together never gets found — the first looks unpromising and is passed over.

Finding the genuinely optimal tree is NP-hard, so every implementation is greedy. It's a practical compromise, and it's one reason a forest of randomised trees beats a single carefully grown one: different trees make different greedy mistakes.

Reading ours

from sklearn.tree import export_text
print(export_text(model.named_steps["clf"], feature_names=list(names)))

|--- prep_min <= -0.2
|    |--- area_Industrial <= 0.5  --> class: 0
|    |--- area_Industrial >  0.5  --> class: 0
|--- prep_min >  -0.2
|    |--- distance_km <= -0.3     --> class: 0
|    |--- distance_km >  -0.3     --> class: 0

It split on prep_min first, then distance_km and area_Industrial — the same columns lesson 18's coefficients ranked highest, found by a completely different mechanism. Two independent methods agreeing about which columns matter is worth more than either alone.

(The thresholds look strange because the pipeline scaled first: -0.2 means 0.2 standard deviations below the mean prep time. Drop the scaler if you want readable thresholds — trees don't need it.)

And every leaf says "on time"

Look again: class 0 in all four leaves. The tree found genuinely informative splits and then predicted the same thing everywhere. Recall at depth 3 is 0.08.

With a 22% positive rate, a leaf has to be more than half late before majority voting says "late" — and none of them is. A leaf that's 40% late is extremely useful information, and majority voting throws it away.

This is lesson 7's threshold problem one level down: voting inside a leaf is a hard-coded 0.5 cut. The fixes are the same too — use predict_proba, which reports the leaf's actual class balance, or reweight the classes. Lesson 22.

What trees are good at

• No scaling needed.
• Mixed column types — numeric and categorical together.
• Interactions for free — every split sits inside the context of the ones above it, so "long trips in heavy rain" is representable without you asking.
• Non-linear thresholds — "over 5 km" is natural rather than approximated.
• Readable, at small depth.

And bad at

• Instability — change a few rows and you get a different tree.
• Memorising at full depth — lesson 9's train 1.000 / test 0.733.
• Extrapolation — predictions are flat beyond the training range, because a leaf just reports what it saw.
• Diagonal boundaries — splits are axis-aligned, so a sloped boundary needs a staircase of them.

The controls are max_depth, min_samples_leaf, min_samples_split and max_leaf_nodes. min_samples_leaf is underrated — it stops the tree building leaves from three rows, which is where memorisation lives.

The turn

Instability sounds fatal and turns out to be exploitable. If trees disagree because of randomness rather than signal, then averaging many of them cancels the noise. That's the next lesson, and it's one of the most valuable ideas in the field.