Handling missing values

Real data often has missing values. Pandas shows a missing value as NaN.

Checking for missing values:

• isnull() marks each value True (missing) or False;
• isnull().any() tells, for each column, whether it has any missing value;
• isnull().sum() counts the missing values in each column;
• isnull().sum().sum() counts them in the whole DataFrame.

Handling missing values:

• dropna() removes every row that has a missing value. This is simple, but you lose the other data in those rows.
• fillna(0) fills each missing value with 0 (or any value you choose).
• ffill() fills a missing value with the value before it. (The book writes this as fillna(method='pad'), which newer pandas has replaced.)

Example: a student's marks in four tests are 20, 22, NaN and 24, out of 25 each. With dropna() we use only the three known tests (66 out of 75, 88%); with fillna(0), 66 out of 100 (66%); with ffill(), the missing mark becomes 22, giving 88 out of 100 (88%).

Each method changes the data, so the results are only approximations. Check first, then drop or fill.