Summary statistics

You already have mean, min, max and sum from lesson 15. This lesson is about the ones that tell you something the mean can't — starting with the most important comparison in descriptive statistics.

mean versus median

The mean uses every value's magnitude, so a single extreme value drags it. The median only cares about order, so it barely notices.

times = np.array([42, 45, 47, 50, 52, 55, 58, 240])
times.mean()        # 73.6  — no request took anywhere near this long
np.median(times)    # 51.0  — a typical request

The mean here describes a request that never happened. Use it when values are roughly symmetric and every one should count; use the median when there are outliers or a long tail — response times, incomes, file sizes, anything where a few values are enormous.

Notice np.median is a function, not a method — there's no x.median(). Same for np.percentile.

Spread

times.std()          # standard deviation
times.var()          # variance — std squared
times.std(ddof=1)    # the sample version

Both are built from squared distances from the mean, which makes them even more outlier-sensitive than the mean. NumPy defaults to ddof=0, the population formula, dividing by n. Statistics courses usually mean ddof=1, dividing by n-1, which is the right choice when your data is a sample from something larger. The difference is small for big n and matters a lot for small.

Percentiles

np.percentile(times, 50)        # the median, by another name
np.percentile(times, [25, 75])  # the quartiles
np.percentile(times, 95)        # the tail — what your slowest users see

The 95th and 99th percentiles are how latency is reported, and for exactly the reason above: an average response time hides the people having a bad time. np.quantile is the same function taking 0–1 instead of 0–100.

The interquartile range — p75 - p25 — is a spread measure that ignores outliers, the way the median does.

Two variables at once

np.corrcoef(a, b)       # a 2x2 matrix; [0, 1] is the correlation
np.cov(a, b)            # covariance, the unnormalised cousin

corrcoef returns a matrix, not a number, because it's built to handle many variables at once. For two arrays you want np.corrcoef(a, b)[0, 1]. The value runs from -1 to 1, and it only measures linear relationship — a perfect parabola can score near zero.

Distributions without plotting

counts, edges = np.histogram(times, bins=5)
np.bincount(labels)          # fast counts for small non-negative ints
np.average(x, weights=w)     # weighted mean

np.histogram gives you the numbers a chart would draw — counts, and the bins + 1 edges that bound them. Handy for checking a distribution before you commit to plotting it, which is the whole matplotlib track away.

All of these take axis, so per-column statistics on a table are one call: cups.mean(axis=0), np.median(cups, axis=0).