4. Missing values

• In a pandas DataFrame, missing values are indicated with NaN, which stands for “not a number”

Detecting missing values

• dogs.isna() # To detect missing values
• dogs.isna().any() # Get the value for each variable that have missing values in that column

Counting missing values

• dogs.isna().sum() # To count the number of NaNs in each column

Plotting missing values

• Use counts to visualize the missing values in the dataset using a bar plot

import matplotlib.pyplot as plt

dogs.isna().sum().plot(kind=“bar”)

plt.show()

Removing missing values

• dogs.dropna()
• Example:

avocados_complete = avocados_2016.dropna() # Remove rows with missing values

print(avocados_complete.isna().any()) # Check if any columns contain missing values

Removing missing values (2)

assert pd.notnull(gapminder.country).all() # Assert that country does not contain any missing values

assert pd.notnull(gapminder.year).all() # Assert that year does not contain any missing values

gapminder = gapminder.dropna() # Drop the missing values

print(gapminder.shape) # Print the shape of gapminder

Replacing missing values

• Replace missing values with another value
• dogs.fillna(0)
• Example:

cols_with_missing = ["small_sold", "large_sold", "xl_sold"] # List the columns with missing values

avocados_2016[cols_with_missing].hist() # Create histograms showing the distributions cols_with_missing

plt.show()

avocados_filled = avocados_2016.fillna(0) # Fill in missing values with 0

avocados_filled[cols_with_missing].hist() # Create histograms of the filled columns

plt.show()