Train and test

Lesson 1 ended on generalisation: the goal is working on rows nobody has seen. This lesson is about the one thing you must do to measure that at all — and it is the single easiest thing in machine learning to get subtly, invisibly wrong.

Grading your own homework

model.fit(X, y)
model.score(X, y)     # 0.94 — and it means nothing

You've asked the model to answer questions it was already given the answers to. Worse, the number rewards exactly the wrong thing: a plain logistic regression scores about 0.81 here, a depth-4 tree 0.85, an unlimited tree 1.00. The model that scores best on the training set is the one that has memorised most, which is to say the one most likely to fail.

Hold some back

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.25,
    stratify=y,
    random_state=0,
)

675 rows to learn from, 225 held back. Note the return order — both X's, then both y's. Getting it wrong doesn't raise; it trains on nonsense and gives you a bad score you can't explain.

The golden rule

The test set is looked at once, at the end.

Not to choose between two models. Not to pick max_depth. Not to decide which features to drop. Every one of those is a decision, and a decision made by looking at the test set is a decision fitted to those 225 rows. Do it twenty times and your test score is as optimistic as a training score — with none of the warning signs, because it's still called a test score.

This is the mistake that ships broken models. It doesn't look like cheating; it looks like diligence. You try things, you check, you keep what works. The fix isn't discipline, it's structure: make those decisions with cross-validation on the training set (lesson 8), and the test set stays clean for the one measurement that matters.

For serious work you'll see a three-way split — train / validation / test. Train fits, validation chooses, test reports. Cross-validation is the same idea with the validation set rotated so you use all your data.

stratify

A plain random split doesn't preserve the class balance, and with an uneven target that matters. Ours is 22.2% late. A random 25% test set comes out at 19.1%:

train_test_split(X, y, test_size=0.25, random_state=0)
# test set is 19.1% late — easier than reality

train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
# test set is 22.2% late — exactly like reality

Three points sounds small. It means you'd be measuring on a test set with fewer late deliveries than the world has, and reporting the result as though it described the world. And it gets worse as the class gets rarer: at 2% positive, an unstratified fold can contain almost no positive rows — or none, at which point precision and recall stop being defined at all.

Stratify on the target for classification. It costs nothing. (For regression there's no direct equivalent — bin the target first if you need it.)

random_state, and how much luck is in your number

The split is random, so without random_state you get a different split — and a different score — every run. Setting it makes your work reproducible. It is not for hunting a seed that flatters you.

Here is why that matters more than it sounds. Same model, same data, eight seeds:

[0.796, 0.796, 0.809, 0.809, 0.836, 0.827, 0.831, 0.827]

Four accuracy points of pure luck, from nothing but which rows landed where. So "our new model is 2% better", measured on one split, is not a finding — it's a statement about the seed. Any single train/test number carries this uncertainty, and reporting it without acknowledging that is the most common overclaim in applied ML.

The fix isn't a better seed. It's taking several measurements and looking at the spread, which is cross-validation, and which is why lesson 8 exists.

How much to hold out

20–25% is the usual answer, and the trade-off is real in both directions. Too small and the score is noisy — 50 test rows means one row is two accuracy points. Too large and you've starved the model in order to measure it.

Scale with what you have: with a million rows, 1% is a fine test set and more would be waste. With 900, 25% gives 225 rows and a score you should already distrust to about ±0.04 — which the seed experiment above just demonstrated. With a few hundred rows, the answer isn't a bigger test set, it's cross-validation.

When a random split is the wrong split

train_test_split assumes your rows are independent. When they aren't, it quietly hands the model a look at the answer:

• Ordered in time. A random split trains on March to predict February. Your score reflects a situation that will never occur. Use TimeSeriesSplit, which always trains on the past and tests on the future.
• Grouped rows. The same customer, patient, or device in both halves. The model can recognise the group rather than learn the pattern. Use GroupKFold or GroupShuffleSplit and split by group.
• Near-duplicates. The same delivery logged twice, landing on both sides. Deduplicate before splitting, not after.

All three are the same failure wearing different clothes: information about a test row was available at training time. That's leakage, and lesson 13 shows the version of it that catches nearly everyone.