Linear algebra

This is the lesson that bridges into machine learning. Every forward pass through a neural network, every linear regression, every rotation of a 3-D model is a matrix product.

* and @ are different operations

a * b    # elementwise — position by position, shapes broadcast
a @ b    # matrix product — rows dotted with columns

One character apart, and mixing them up is the most common linear-algebra bug in Python. It's especially easy coming from MATLAB, where * means the matrix product and .* means elementwise — the exact opposite convention.

The tell is usually the shape. If a * b gives you something the same shape as your inputs when you expected it to shrink, you wanted @.

How the matrix product works

Each output cell is one row of the left dotted with one column of the right — multiply the pairs, add them up:

result[0, 0] = 1*7 + 2*9 + 3*11 = 58
result[0, 1] = 1*8 + 2*10 + 3*12 = 64

Which gives the shape rule: (m, n) @ (n, p) → (m, p). The inner dimensions must match — they're the length of the row and the column being dotted — and they disappear from the result.

(2,3) @ (3,2) -> (2,2)     ok
(2,3) @ (2,3) -> ValueError: matmul: core dimension mismatch

Half the transposes in machine-learning code exist to make that inner pair line up.

Matrix times vector

A @ v      # (2,2) @ (2,) -> (2,)

A 1-D array is treated as whichever orientation makes the product work, and the result comes back 1-D. That's the shape of a single prediction: weights times features.

Solving, and the thing not to do

For a system Ax = b, algebra says x = A⁻¹b. Don't write it that way:

np.linalg.solve(A, b)        # do this
np.linalg.inv(A) @ b         # same answer, slower and less accurate

solve factorises the matrix directly rather than computing a full inverse and then multiplying. It's faster, and it doesn't throw away precision on a step you didn't need. The rule in numerical work is: if you're calling inv, you probably wanted solve.

When there's no exact solution — more equations than unknowns, which is what fitting a line to scattered points looks like — np.linalg.lstsq finds the best compromise. That's linear regression, in one call.

The rest of the toolbox

np.linalg.norm(v)          # length of a vector
np.linalg.norm(a - b)      # distance between two points
np.linalg.det(A)           # zero means no unique solution
np.linalg.matrix_rank(A)   # how many independent directions
np.linalg.eig(A)           # eigenvalues and eigenvectors — PCA
np.linalg.svd(A)           # singular value decomposition

norm is the one you'll use most, and not for anything exotic — np.linalg.norm(a - b) is the Euclidean distance between two points, which is the heart of k-nearest-neighbours and k-means.

dot, matmul and @

For 2-D arrays, a @ b, np.matmul(a, b) and np.dot(a, b) are identical. For higher dimensions they diverge: matmul broadcasts the leading axes and multiplies the last two — which is what "a batch of matrices" needs — while dot does something else entirely. Use @ and you'll never have to think about it.