query, isin and between
The readable ways to say a filter, once masks stop scaling.
.query().isin().between().str.contains()@variables~Watch it happen
Play it through, or step back and forth yourself.
ordersdf[(df["cups"] > 100) & (df["city"] == "Delhi") & (df["rating"] > 4)]
Boolean masks work, and they get unwieldy fast. Three conditions and the frame's name appears six times — df[(df["a"] > 1) & (df["b"] < 2) & ...]. It's correct and it's hard to read.
The idea
Boolean masks are correct and they don't scale to reading well. Three conditions and the frame's name appears six times:
orders[(orders["cups"] > 100) & (orders["city"] == "Delhi") & (orders["rating"] > 4)]query
.query() takes the condition as a string, with the column names bare:
orders.query("cups > 100")
orders.query("cups > 100 and city == 'Delhi' and rating > 4")Note you can use and, or and not here. Inside the string they're safe — pandas parses the expression itself rather than letting Python evaluate it, so the ambiguity that forces & in a mask doesn't arise.
Reference a Python variable with @:
threshold = 100
orders.query("cups > @threshold")
orders.query("city in @wanted")A column name with a space in it needs backticks — orders.query("`unit price` > 20").
isin and between
Two helpers that collapse long chains:
orders[orders["city"].isin(["Delhi", "Pune"])] # instead of a stack of |
orders[orders["cups"].between(90, 130)] # inclusive at BOTH ends
orders[orders["cups"].between(90, 130, inclusive="neither")]
orders[~orders["city"].isin(["Delhi"])] # ~ negates any maskbetween including both ends is the opposite of a Python slice, so it's worth checking when a boundary value matters.
Text conditions
The .str accessor gives you vectorised versions of the usual string methods, each returning a mask:
orders[orders["item"].str.contains("ginger")]
orders[orders["item"].str.contains("ginger", case=False)] # catches "Ginger"
orders[orders["city"].str.startswith("M")]
orders[orders["item"].str.match(r"^gin")] # regexWatch out for missing values: .str.contains returns NaN for them, which then can't be used as a mask. Pass na=False to treat gaps as "no match", which is nearly always what you want.
Which to use
Masks for one or two conditions, and any time you want the mask itself — to count with .sum(), or reuse it. query when there are several and readability wins.
Two honest downsides to query: it's a little slower, since it parses a string on every call; and a typo in a column name isn't caught until the line runs, whereas a mask fails where you wrote it.
See it run
The lesson's code, ready to run and to fiddle with.
Putting the kettle on…
Starting up…
Worked example
not gradedAlready written and ready to go — press Run to see what it does, then change a number, a column name, anything, and run it again.
tryDrop case=False from the last line and see how many rows you lose.
Your turn
4 exercises. Write the code yourself, then press Check — a nudge and the answer are there if you want them.
Use .query() to keep the rows where cups is over 100.
Now rows from Delhi with more than 100 cups, using .query().
Keep the rows whose city is Delhi or Pune, using .isin().
Keep every row whose item contains "ginger", ignoring case — the data has both ginger and Ginger.
