Categories and memory

A city column stores the string "Delhi" once per row. Over a million rows that's a million copies of the same handful of words, and every comparison is a string comparison.

orders["city"] = orders["city"].astype("category")

category stores the distinct values once in a lookup and keeps one small integer per row. The column still behaves like text — you can still compare it, group by it, print it — but underneath it's int8.

The memory

orders.memory_usage(deep=True)

deep=True matters. Without it you're measuring the pointers, not the strings they point at, and a text column looks suspiciously cheap.

Ten-fold reductions are ordinary when values repeat a lot. The rule of thumb: if the number of distinct values is a small fraction of the number of rows, it's worth it. If nearly every value is unique — an id, a free-text note — a category is bigger, because you're storing the lookup and the codes.

Grouping and joining get faster too, since they become integer operations.

The other reason: ordering

This one changes how code reads. Sort low, medium, high as text and you get high, low, medium — alphabetical, and wrong in a way that looks like a data problem.

band = pd.CategoricalDtype(["low", "medium", "high"], ordered=True)
df["band"] = df["band"].astype(band)

df.sort_values("band")     # low, medium, high
df["band"] > "low"          # works — comparison respects the order
df["band"].max()            # "high"

Declare the order once and sorting, comparison, grouping and plotting all respect it. pd.cut from lesson 8 returns an ordered category for exactly this reason.

The .cat accessor

s.cat.categories              # the lookup
s.cat.codes                    # the integer per row
s.cat.add_categories(["Jaipur"])
s.cat.remove_unused_categories()
s.cat.rename_categories({"low": "L"})

Two things that will catch you

Assigning an unknown value raises. Setting a city to "Jaipur" when it isn't in the categories is an error, not an addition. Call .cat.add_categories() first. That strictness is a feature — it catches typos that a plain text column would swallow.

groupby keeps empty categories. Grouping on a category returns a row for every category, including ones with no matching rows — which is either exactly what you wanted for a complete report, or a pile of zeros you didn't ask for. Pass observed=True for only the ones present.

Related: after filtering, the categories that no longer appear are still in the lookup, which is why value_counts() can show zeros. remove_unused_categories() tidies up.

Where to do it

On the way in, once the text is clean — normalise first, or ginger and Ginger become two separate categories and you've preserved the bug in a more efficient format:

orders["item"] = orders["item"].str.strip().str.lower().astype("category")
pd.read_csv(f, dtype={"city": "category"})