This step is really important before going on any further analysis. Data preparation or preprocessing is a big issue for both data warehousing and data mining.

Indeed in every dataset, we could find some data which will be really dirty, because for many different reasons it could be :

• incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data
• noisy: containing errors or outliers
• inconsistent: containing discrepancies in codes or names
• duplicate records

So the main purpose will be to determine whether or not the data is dirty, and if it is to do data pre-processing in order to correct that problem.

Because if there is no quality data, there will be no quality mining results! And so on there will be no quality decisions.

Nowadays, a lot a methods have been developed but data preprocessing remains an active area of research.

1) Label Encoding

What is a label encoding and why we use it?

We use the label encoding when we want our data to be an understandable numerical data. When we want to use a prediction model, our dataset must have only numeric value. For exemple if we look, the first column is the column wherewe can find all text. So we have to transform this column into a numerical data if we wish to use it in our prediction.

How applied it to our dataset?

We got 3 no-numerical column in our dataset if we exclude the last column which will be our target. To transform these columns we imported in python the label encoding method from the library Scikit-Learn.

2) Data Normalization

- What is the data normalization ?

First of all we use the data normalization to do a scaled to fall within a small, specified range. But we can use it also when we want all features to contribute equally in our models.

- How can we normalize our data?

There are many methods to do this :

- Min-max normalization

- Z-Score normalization

- Normalization by decimal scaling

Let's try the 2 firsts methods.

- Min-Max Normalization

With this technique we transform our data by using this formula :

y= (x-min)/(max-min)

Where 'x' is the value to transform, 'min' the smalest value of the column and 'max' the column.

For our dataset we created a function to applied the Min-Max Normalization to each column. Inside the function we imported a method from Scikit-Learn library which execute automatically the Min-Max formula (5th line). We just had to precise which column we want to numerize and the desired range of transformed data (here between 0 and 1).

Below, a screenshot of our dataset with all the values normalized. We can see that all the values are between 0 and 1 as planished.

- Z-Score Normalization

The Z-score allow us to transform the data onto a distribution whose mean is defined as 0 and whose standard deviation is defined as 1.

With this technique we transform our data by using this formula :

Z = (x-µ )/σ

Where 'x' is the value to transform, 'µ' the mean of the column and 'σ ' the standard deviation of the column.

Like before, we created a function to applied the Z-Score Normalization to each column. Inside the function we just calculate the Z-Score by using the formula above.

Below, a screenshot of our dataset with all the values after the Z-Score normalization. The purpose was to have all the values near to 0 (usually between -3 and 3).

In this example, we tried to discretize the "Customer Service Calls" column by dividing into 3 values categories (rare, sometimes, often) based on the number of customer service calls. For this exemple and the next one, we will use a normalize dataset.

As we cann see our numerical data has been transform into a discret attributes. Those discretizes data are very important because their are essential for some of our analysis like the association rules.