Pipelines and leakage

Add minutes to the feature list and watch what happens:

honest features    5-fold AUC 0.8590
+ minutes          5-fold AUC 0.9989

Nearly perfect. If a number like that has ever produced a flash of delight rather than suspicion, this is the lesson to remember — because in practice it almost always means a mistake rather than a breakthrough.

Because late IS minutes

late = (minutes > 30)

The model didn't learn anything about deliveries. It learned to compare a number to 30. And at prediction time you will not have minutes, because the delivery hasn't happened yet — which is the entire reason you're predicting.

The general rule

Leakage is any information available at training time that will not be available at prediction time.

It always looks like success, and that's what makes it dangerous — every incentive you have points towards believing it. It is the most common reason a model that tested beautifully collapses in production.

Where it hides

Ours is cartoonishly obvious. Real ones rarely are:

• A column recorded after the outcome. resolution_time, refund_issued, minutes.
• An ID that encodes the answer. Case numbers assigned in outcome order are a classic.
• A status field updated later. customer_status = "churned" predicts churn perfectly.
• Aggregates over all rows. A per-city mean computed before splitting includes the test rows.
• Duplicates across the split, and time-ordered rows split randomly — lesson 3's warnings, which were leakage all along.

The question that catches most of them: for each feature, when does its value come into existence? If the answer is "after the thing I'm predicting", it cannot be a feature.

The quiet leak

There's a second kind that needs no bad column at all:

X_scaled = StandardScaler().fit_transform(X)     # all 900 rows
cross_val_score(model, X_scaled, y, cv=5)

The scaler's mean and standard deviation were computed from rows that every fold then used as its test set. There's no obviously wrong column here — just a statistic that saw data it shouldn't have.

The inflation is small, a fraction of a point, which is exactly why it survives review. It is a consistent, invisible thumb on the scale, and it applies to every fitted preprocessor: scalers, imputers, encoders, feature selectors, PCA. Anything with a trailing underscore learned something, and where it learned it from matters.

And it's worse under cross-validation than under a single split, because the test fold changes each round. Preprocess once up front and you've leaked into all five — while using the very tool you adopted in order to be careful.

Pipeline

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

num = Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
                ("scale", StandardScaler())])

pre = ColumnTransformer([("num", num, NUMERIC),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), CATEGORICAL)])

model = Pipeline([("pre", pre),
                  ("clf", LogisticRegression(max_iter=1000))])

cross_val_score(model, X, y, cv=5, scoring="roc_auc")     # honest

Every step is fitted on the training rows of whatever fold it's in, then applied to that fold's test rows. You are no longer relying on remembering the rule — the structure removes the place where the mistake could be made.

A pipeline is an estimator

This is what makes it more than a tidiness device. A Pipeline has fit, predict and get_params, so it goes anywhere an estimator goes:

cross_val_score(model, X, y, cv=5)          # safe
GridSearchCV(model, {"clf__C": [0.1, 1]})    # safe — module 6
joblib.dump(model, "model.joblib")           # preprocessing ships with the model

model.named_steps["clf"].coef_               # reach inside
model.set_params(clf__C=0.1)                  # step__param

That last convention — double underscore to address a parameter inside a step, and again for a step inside a nested pipeline (pre__num__impute__strategy) — is what module 6 uses to tune everything at once.

The deployment argument matters too: pickling the pipeline means the preprocessing and the model can never drift out of sync. A model saved without its scaler is a bug waiting for a deploy.

Two conveniences

make_pipeline(SimpleImputer(), StandardScaler(), LogisticRegression())
# auto-names the steps: "simpleimputer", "standardscaler", "logisticregression"

make_column_selector(dtype_include="number")
# picks columns by dtype instead of by name

make_pipeline is shorter; explicit Pipeline names are better when you'll refer to a step in a parameter grid, because clf__C beats logisticregression__C.

Where we are

lesson 4    numeric only              0.8142
lesson 11   + categoricals            0.8590
lesson 12   + missingness indicator   0.8786

All of it from preparing the data properly — not one change of algorithm. That ordering is the usual one in practice, and it's why this module comes before the model modules rather than after.