Data cleaning is a crucial step in the data preparation process, involving the identification and correction of errors, inconsistencies, and missing values within a dataset. It ensures that the data is accurate, reliable, and suitable for analysis or machine learning applications.
Embedded Files
In the dataset collected, there were some null values in the text column which is practically the most important column for our problem statement. Since, the number of rows with the news text column are quite less (46 out of 2096 rows), they were removed from the data set.
After this step, before analyzing the data, it is important to understand what columns are actually important for the analysis. The columns containing site and image URLs are not relevant as of now. Hence, those columns were removed.
Once this is done, the ‘published’ column that contains the information about the date and time of when the news was published needs to be cleaned. The date and time of news publication can be relevant factors when identifying whether news is real or fake. This is due to the possibility of temporal patterns, news cycle influence, event correlation, social media impact and so on. The dataset after this step has two separate columns - date and time.
Lastly, before proceeding to EDA, it is essential to check for any incorrect values, irregular expressions or special characters that might have been in the data (especially in the textual data). There are some columns that do have such special characters. The author column has ‘@’ as a special character that is occasionally mentioned in their email id. In the title column, some words have ` symbol on top of some letters to emphasize. The ‘text’ column has ‘\n’ repeated various times which has to be taken care of. These minor issues were handled with the help of regular expressions.
Page updated
Report abuse