Regression metrics

Regression errors come in the units of the thing you predicted, which makes them far easier to talk about than accuracy. Ours are minutes, and a stall manager can tell you immediately whether three minutes matters.

The four numbers

MAE    3.1019     mean |error|          every miss counts the same
RMSE   3.8605     sqrt(mean error²)     big misses count more
MedAE  2.5709     median |error|        outliers ignored entirely
R²     0.5948     variance explained    the mean scores exactly 0

MAE is the honest average miss. Quote this to people: it's in minutes and it means precisely what it sounds like. It's robust to outliers, and it is not what least squares optimises.

RMSE squares the errors first, so one 10-minute miss contributes as much as four 5-minute ones. It's always ≥ MAE, and it is what LinearRegression minimises — so the model is already tuned for it. Use it when large errors are disproportionately bad, which for deliveries they are.

Their ratio is a free diagnostic

RMSE / MAE = 3.8605 / 3.1019 = 1.245

Close to 1.0 means your errors are all about the same size. 2 or more means a handful of large misses are carrying the RMSE, and you should go and look at them individually.

Ours is 1.245 — mild. Comparing MAE 3.10 with MedAE 2.57 tells the same story from the other side: the mean is pulled up a little by a few bad rows, and the worst single miss is 9.9 minutes.

R², and its two edges

R²'s virtue is that its baseline is built in: predicting the mean scores exactly 0. Unlike accuracy, you can read 0.59 without knowing the units, the spread, or anything else about the data.

Its weakness is the mirror image — it tells you nothing about whether the errors are acceptable. An R² of 0.59 with a MAE of 3 minutes is fine for chai deliveries and catastrophic for anaesthesia dosing. Always report R² and an error in real units.

R² can be negative, and it isn't a bug

perfect                  1.00
our linear model         0.59
predicting the mean      0.00
degree-3 polynomial     −2.11

Negative means worse than a horizontal line through the mean. It has no lower bound. This catches people constantly — the name implies something squared and therefore positive, but out of sample it isn't.

It happens more often than you'd expect: a badly overfitted model, a test set the model wasn't built for, or a transform that went wrong. If you see it, don't hunt for the bug in your metric code — the model really is that bad.

A word on MAPE

mean_absolute_percentage_error is tempting because percentages feel comparable across problems. Two warnings: it's undefined when the true value is zero, and it punishes over-prediction more than under-prediction, which quietly biases any model you tune on it. Use it only when your target is strictly positive and a percentage is genuinely what the reader wants.

The residual plot

No single number tells you whether the model is right. For that you plot:

resid = y_test - model.predict(X_test)
ax.scatter(model.predict(X_test), resid, alpha=0.5)
ax.axhline(0, linestyle="--")

Residual against prediction, not against the true value — plotting against the truth produces a spurious diagonal that has fooled a great many people.

What you want is a shapeless band around zero. Every pattern means something specific:

• A curve → the relationship isn't linear. Add polynomial terms or use a tree-based model.
• A fan (spread grows with the prediction) → non-constant error. Try modelling log(y).
• Drift off zero in a region → the model is biased for a subgroup. Go and find which one.
• Isolated far-out points → outliers worth investigating individually rather than averaging away.

Every one of those is an assumption from lesson 14 being violated, and each has a standard fix. This is the regression equivalent of the confusion matrix: the summary number tells you how much you're wrong, and the plot tells you how.

Ours

residual mean  −0.0172        std  3.8691        worst  9.93 minutes

A flat band, centred essentially on zero. The straight-line assumption holds — which is exactly why the random forest in lesson 14 couldn't beat it, and a nice piece of corroboration between two completely different diagnostics.