Missing data

Our orders table has three gaps — two missing ratings and one missing revenue. That's deliberate — real data always has gaps, and how you handle them changes your results more than almost anything else you'll do.

Find them first

orders.isna()                    # a frame of booleans, same shape
orders.isna().sum()              # how many per column
orders.isna().mean()             # what FRACTION per column
orders.isna().sum().sum()        # 3 — the total
orders[orders["rating"].isna()]  # the offending rows themselves

.isna() and .isnull() are the same function with two names, as are .notna() and .notnull(). Use whichever reads better; most code uses isna.

The fraction is usually the more useful number. Two missing values out of twelve is a different problem from two out of two million.

Dropping

orders.dropna()                      # ANY gap in ANY column — 10 of 13 rows
orders.dropna(subset=["rating"])     # 11 rows — only the rating gaps
orders.dropna(how="all")             # only rows that are entirely empty
orders.dropna(thresh=5)              # keep rows with 5+ real values
orders.dropna(axis=1)                # drop the COLUMN instead of the rows

A bare dropna() is more aggressive than people expect — one gap anywhere on the row and it's gone. On a wide table with scattered gaps it can remove nearly everything. Always pass subset= unless you genuinely mean "any column".

And check what you lost: len(before) - len(after). If that number is large, the gaps are telling you something and deleting them is throwing the message away.

Filling

orders.fillna(0)                              # a constant
orders.fillna({"rating": 0, "cups": 0})       # per column
orders["rating"].fillna(orders["rating"].mean())   # the column's mean
orders["rating"].ffill()                      # carry the last value forward
orders["rating"].bfill()                      # pull the next one back

ffill and bfill only make sense when the rows are ordered — a time series, a sorted log. On unordered rows, "the previous value" is meaningless and you're inventing data from whatever order the file happened to be in.

Filling within groups is often what you actually want: orders.groupby("city")["rating"].transform("mean") gives each city's own average rather than one global number. That's Module 4.

The part that matters

The same column, three defensible treatments, three different answers:

orders["rating"].mean()               # 4.24  — skips the gaps (the default)
orders["rating"].fillna(0).mean()     # 3.58  — treats missing as zero
orders["rating"].ffill().mean()       # 4.19  — carries the previous forward

None of those is wrong. But only one matches what you meant, and the difference between 4.24 and 3.58 is the difference between a good month and a bad one. Decide deliberately, and leave a comment saying why.

Worth noting: pandas' default is to skip missing values in aggregations. That's a choice too — it's why mean() gives 4.24 rather than NaN, unlike NumPy.

NaN, None and pd.NA

Three spellings of "missing", for historical reasons. np.nan is a float and lives in float columns. None is Python's null, and pandas converts it on the way in. pd.NA is the newer one used by the nullable dtypes from lesson 3.

You rarely need to care which you have, as long as you use .isna() to test rather than == None or == np.nan — neither of which works.