Digits — where kNN finally wins

1,797 handwritten digits, each an 8×8 greyscale image with pixel values from 0 to 16. It's a miniature of MNIST, and it fits in memory instantly.

digits = load_digits()
digits.images.shape    # (1797, 8, 8)   as pictures
digits.data.shape      # (1797, 64)     flattened
digits.data.max()      # 16.0

Flattening is the trick

Take the 8×8 grid and lay it out in a row: 64 columns, one per pixel. Now an ordinary tabular model can see pictures. Nothing about LogisticRegression knows it's looking at an image — it just has 64 numbers.

And it costs something real. The model has no idea that pixel 9 sits below pixel 1. Shuffle all 64 columns — the same shuffle for every row — and it scores identically.

That's the gap convolutional networks exist to fill: they keep the spatial structure the flattening throws away. For 8×8 digits it barely matters; for a photograph it matters enormously.

Ten classes, and a low floor

Roughly 180 examples each, so the dummy scores about 0.10. There is a lot of room above the baseline here — unlike the chai data, where the floor was 0.778 and the best model reached 0.879. That's what a problem with genuine signal looks like.

kNN finally wins

kNN (k=3)             0.9449 ± 0.0173
random forest (200)   0.9366 ± 0.0239
logistic regression   0.9204 ± 0.0300

First time in this track. In lesson 19 kNN lost to logistic regression by three points; in lesson 21 every ensemble lost too. Here the order inverts completely.

And the reason is worth understanding

All 64 columns are the same kind of thing, measured the same way — ink, on a 0-16 scale. So the Euclidean distance between two rows is a genuine measure of how similar two images look, and "find the three most similar digits and take a vote" is a sensible algorithm.

On the chai data, distance meant adding kilometres to degrees Celsius to counts of items. Scaling papers over the units; it cannot make the sum mean anything.

This is the real argument against having a favourite algorithm. The right model is a property of the data — same library, same code, different problem, opposite answer.

A 10×10 confusion matrix

Ten classes means a hundred cells, and everything interesting is off the diagonal. The confusions are 3↔2, 3↔8, 8↔1 and 9↔3.

Look at those pairs and squint: at 8×8 resolution a 3 and an 8 really do differ by a couple of pixels on the left edge. The errors are the ones a human would make too, which is a good sign — it means the model has learned something about shape rather than about artefacts.

This is the confusion matrix earning its place from lesson 6. An accuracy of 0.945 tells you how often; the matrix tells you which — and "which" is what you can act on. If 3s and 8s were costing you money, you'd know exactly where to look.

PCA that actually pays

31 of 64 components reach 90% of the variance

kNN on all 64 columns    0.9449
kNN on 20 components     0.9310

A third of the columns for 1.4 points. That's a real trade — and on the chai data (lesson 27) PCA was pure loss, costing 0.07 AUC for nothing.

The difference is redundancy. Neighbouring pixels are strongly correlated, and the corner pixels are blank in almost every image, so there genuinely is structure to compress. Lesson 27 said PCA earns its place on "hundreds of correlated columns" — this is what that looks like when it's true.

It also makes kNN meaningfully faster, which matters for a model that scans the whole training set on every prediction.

What this dataset is good for

• Practising multi-class work at a size that stays instant.
• Seeing why distance-based models sometimes win.
• Reading a large confusion matrix and acting on it.
• Watching PCA behave the way the textbooks describe.

And it's an honest warm-up for image work: everything here scales up to MNIST's 70,000 28×28 digits, where the same kNN idea still works and simply becomes slow.