Decision boundaries and kNN

Every classifier draws a decision boundary — the surface where the predicted class flips. Model families differ mostly in what shapes they're allowed to draw, and that's the most useful way to keep them straight in your head.

Logistic regression draws one straight cut. That's its entire hypothesis space: very stable, and completely unable to represent a class that occupies two separate regions. That limitation is its bias, in lesson 16's sense.

k-nearest neighbours draws anything

from sklearn.neighbors import KNeighborsClassifier
KNeighborsClassifier(n_neighbors=25)

kNN has no boundary of its own. To classify a row it finds the k closest training rows and takes a vote. The boundary is whatever that produces — as wiggly as the data demands.

It doesn't really fit

fit stores the training data. That's all. Every bit of work happens at predict, which makes kNN instant to train and slow to use — the exact inverse of everything else in this track. It's sometimes called a lazy learner, and with a million training rows every single prediction has to consult all of them.

The dial runs backwards

This is the part worth internalising, because it's the opposite of every hyperparameter so far. Small k is the flexible end.

k       train      5-fold AUC
  1     1.0000        0.6543
  5     0.8785        0.7790
 15     0.8252        0.8206
 25     0.8104        0.8398
 50     0.7985        0.8404
100     0.7822        0.8456

At k=1 the training score is exactly 1.0000 — and of course it is, because the nearest neighbour of a training row is itself. That is lesson 1's lookup table, arrived at by an ordinary algorithm rather than a strawman. Its cross-validated AUC is 0.6543, barely better than a coin flip in ranking terms.

As k rises each prediction averages more neighbours, the boundary smooths, and generalisation improves the whole way. So "increase max_depth for more capacity" and "increase n_neighbors for more capacity" point in opposite directions — reading it backwards is a very common way to make a model worse while trying to fix it.

Notice the curve is still climbing at k=100. We haven't found the peak, which is a hint to search further — and a reminder to look at the shape of a sweep, not just the best value in your list.

What kNN needs

• Scaling, absolutely. Distance is the model, so an unscaled column dominates it. Lesson 10 measured this at +0.032 AUC — the largest scaling effect in the track.
• Not too many columns. In high dimensions every point is roughly equidistant from every other and "nearest" stops meaning anything. This is the curse of dimensionality, and our one-hot encoding alone took us to 18 columns.
• Patience at predict time, or an approximate index.

And there's nothing to read afterwards — no coefficients, no feature importances. You need permutation importance (lesson 25) to learn anything about which columns mattered.

How it does

best kNN (k=100)       0.8456
logistic regression    0.8786

The flexible boundary simply wasn't needed — the real one is close to a straight cut, which is what module 4 concluded from the regression side too.

That's still a useful six lines. Fitting a very different model family and losing tells you something true about the shape of your problem, and that's worth knowing even when you don't ship the model.

A note on boundaries generally

Keeping the families straight by what they can draw:

• Logistic regression, linear SVM — one straight cut.
• Decision tree — axis-aligned rectangles. A diagonal needs a staircase.
• Random forest, boosting — many rectangles, averaged into something smoother.
• kNN, RBF SVM — arbitrarily curved.

When a model underperforms, asking "can it even draw the shape this problem needs?" is usually more productive than tuning it.