Missing data

What is missing data

• Some tabular data just don't have value(Nan in python)

• Using other expressions like non, don't have value, 9999 etc

Why is missing

• Can't collect

• have problems when fetching data

• just random missing

How to deal with missing

Typically, we just want to get rid of the missing data for the analysis step. Usually just drop data is the most convient way, but it will lose useful information.

There are still some methods can be used to deal with the missing data, here introduce in data type.

Sometimes the missing data isn't just show NaN, So you should really be familer with the dataset

Categorical variables missing

• drop it if small

• fill na with mode ( it also can be used in numeric variables, just more helps in categorical data)

Numeric Variables missing

• just drop it if small

• fill na with mean or median based on the distribution

• fill na using regressions or other methods

• if it's time series data, the previous, last or moving average could be useful. But I will not include this part in the code example.

Change Data Type

Basic use data type in python

• string/category

• int

• float

• boolean

• Time Stamp

Common Errors in dataset

• Some ranking data was recognized as numeric data

• Some numeric data have type issue and can not be recognized as

• identify the category variables

Functions

• pandas

• merge

• Pivot and pivot table

• melt

• stack and unstack

• Groupby

animation helps to understand

Code Example

missing data

change data type