agg, transform and apply

Once you have a GroupBy, there are three genuinely different things you might want from it. They're told apart by the shape of what comes back, and that's the only distinction worth memorising.

g = orders.groupby("city")

g.agg("sum")          # one row per GROUP        — 3 rows from 13
g.transform("sum")    # one value per ROW        — 13 rows from 13
g.filter(fn)          # whole groups kept/dropped — a subset of the rows

agg — collapse

This is the groupby you already know. The useful form is the named one:

g.agg(
    total=("cups", "sum"),
    best=("cups", "max"),
    n_orders=("cups", "count"),
)

You get the column names you chose. The alternatives — g.agg(["sum", "max"]) and g.agg({"cups": "sum"}) — both work, but the first gives you a MultiIndex of columns to untangle afterwards. Named aggregation avoids that entirely, and it's the form to teach from.

You can pass your own function too, as long as it reduces a Series to one value: g.agg(spread=("cups", lambda s: s.max() - s.min())).

transform — broadcast back

transform computes the same group summaries but returns one value per original row, aligned to the original index. Every Delhi row gets Delhi's total.

That's the thing agg cannot do, and it's what you need whenever a row's answer depends on its group:

# each row's share of its city's total
orders["share"] = orders["cups"] / g["cups"].transform("sum")

# distance from the city's average
orders["vs_avg"] = orders["cups"] - g["cups"].transform("mean")

# fill gaps with the CITY's mean rating, not one global number
orders["rating"] = orders["rating"].fillna(g["rating"].transform("mean"))

That last one is the payoff from lesson 9. Filling a gap with the overall mean throws away the fact that different cities score differently; filling with the group's own mean doesn't.

The window functions work per group too: g["cups"].rank(), g["cups"].cumsum(), g["cups"].shift() — each restarts at every group boundary.

filter — keep or drop whole groups

g.filter(lambda x: x["cups"].sum() > 200)

The function receives each group as a DataFrame and returns True or False. Groups that pass keep all their rows; groups that fail lose all of theirs.

This is not row filtering, and the difference matters. "Cities that sold more than 200 cups in total" is a statement about groups; you can't express it with a row mask.

apply — the escape hatch

g.apply(fn) hands each group to your function and assembles whatever comes back. It can do anything, and that's both the appeal and the problem: it runs a Python loop over the groups, so on many groups it's dramatically slower than the built-ins.

Reach for it when you genuinely need per-group logic that agg and transform can't express — the top 2 rows of each group, say — and reach for the others first the rest of the time.

One small thing

Grouping puts the keys in the index. If you'd rather they stayed a column — for a merge, or for plotting — say so up front:

orders.groupby("city", as_index=False)["cups"].sum()
orders.groupby("city")["cups"].sum().reset_index()   # same thing, after the fact

And dropna=False if you want a group for the missing keys, which pandas discards by default — quietly, and that has surprised people out of a chunk of their data.