Check for linearity

Before you execute a linear regression model, it is advisable to validate that certain assumptions are met. As noted earlier, you may want to check that a linear relationship exists between the dependent variable and each independent variable. To perform a quick linearity check, you can use scatter diagrams (utilizing the Seaborn library) [2]. 

The next numerical example will explore a linear relationship exists between the [3]:

• index_price (dependent variable) and interest_rate (independent variable),

• index_price (dependent variable) and unemployment_rate (independent variable).

Let's define the data for the numerical example.

import pandas as pd

data = {'year': [2017,2017,2017,2017,2017,2017,2017,2017,2017,2017,2017,2017,2016,2016,2016,2016,2016,2016,2016,2016,2016,2016,2016,2016],

        'month': [12,11,10,9,8,7,6,5,4,3,2,1,12,11,10,9,8,7,6,5,4,3,2,1],

        'interest_rate': [2.75,2.5,2.5,2.5,2.5,2.5,2.5,2.25,2.25,2.25,2,2,2,1.75,1.75,1.75,1.75,1.75,1.75,1.75,1.75,1.75,1.75,1.75],

        'unemployment_rate': [5.3,5.3,5.3,5.3,5.4,5.6,5.5,5.5,5.5,5.6,5.7,5.9,6,5.9,5.8,6.1,6.2,6.1,6.1,6.1,5.9,6.2,6.2,6.1],

        'index_price': [1464,1394,1357,1293,1256,1254,1234,1195,1159,1167,1130,1075,1047,965,943,958,971,949,884,866,876,822,704,719]        

        }

df = pd.DataFrame(data)

df

The next code automatically produces scatter plots among several variables organized as a matrix of experiments.

import seaborn as sns

# Selection: interest_rate  unemployment_rate  index_price

X = df[list(df.columns)[2:]]

sns.pairplot(X);

The last row of graphics it possible to extract the following conclusions:

• A linear relationship exists between the index_price and the interest_rate. Specifically, when interest rates go up, the index price also goes up.

• A linear relationship also exists between the index_price and the unemployment_rate – when the unemployment rates go up, the index price goes down (here we still have a linear relationship but with a negative slope).

• The values of index_price follow a uniform distribution.

Building the model: coefficients

The next step is to obtain the model coefficients and give an interpretation of them. This is done in the next code.

import pandas as pd

from sklearn import linear_model

import statsmodels.api as sm

x = df[['interest_rate','unemployment_rate']]

y = df['index_price']

# with sklearn

regr = linear_model.LinearRegression()

regr.fit(x, y)

print('Intercept: \n', regr.intercept_)

print('Coefficients: \n', regr.coef_)

Intercept: 1798.4039776258544 

Coefficients: [ 345.54008701 -250.14657137] 

This output includes the intercept and coefficients. You can use this information to build the multiple linear regression equation as follows:

index_price = (intercept) + (interest_rate coef)*X1 + (unemployment_rate coef)*X2

And once you plug the numbers:

index_price = (1798.4040) + (345.5401)*X1 + (-250.1466)*X2

Building the model: R squared, Adjusted R squared, and P-value

Model predictions and graphical performance evaluation

Now, the model could be employed to predict index_price using interest_rate_coef and unemployment_rate_coef and show the prediction and real data in a graphic.

import matplotlib.pyplot as plt

y_hat = model.predict(xm)

xp = list(range(1,len(x)+1))

plt.plot(xp,y,'ob',label='Data')

plt.plot(xp,y,'-r')

plt.plot(xp,y_hat,'--g',label='Prediction')

plt.xlabel('Time')

plt.ylabel('index_price')

plt.legend()

plt.grid()

plt.show()

Model predictions and metrics for performance evaluation

For linear regression models there are three metrics for performance evaluation which are [4]:

• Mean Absolute Error (MAE): Mean Absolute Error is the absolute difference between the actual or true values and the predicted values. The lower the value, the better the model’s performance. A mean absolute error of 0 means that your model is a perfect predictor of the outputs. 

• Mean Square Error (MSE): Mean Square Error is calculated by taking the average of the square of the difference between the original and predicted values of the data. The lower the value, the better the model’s performance.  

• Root Mean Square Error (RMSE): Root Mean Square Error is the standard deviation of the errors that occur when a prediction is made on a dataset. This is the same as the Mean Squared Error, but the root of the value is considered while determining the accuracy of the model. The lower the value, the better the model’s performance.

In mathematical terms, we have [5]:

#Model Evaluation

from sklearn import metrics

import numpy as np

meanAbErr = metrics.mean_absolute_error(y, y_hat)

meanSqErr = metrics.mean_squared_error(y, y_hat)

rootMeanSqErr = np.sqrt(metrics.mean_squared_error(y, y_hat))

print('Mean Absolute Error:', meanAbErr)

print('Mean Square Error:', meanSqErr)

print('Root Mean Square Error:', rootMeanSqErr)

Mean Absolute Error: 51.77251657344119 

Mean Square Error: 4356.611357123128 

Root Mean Square Error: 66.00463133086289 

All three previous metrics are important to compare different linear regression models. For example, to verify if the inclusion of an additional variable could improve or not the model performance.

The Python code with all the steps is summarized in this Google Colab (click on the link):

https://colab.research.google.com/drive/1UCEYruS4At3frlJPxYhAXQxrN5CBNuls?usp=sharing

References

[1] https://medium.com/machine-learning-with-python/multiple-linear-regression-implementation-in-python-2de9b303fc0c

[2] https://www.sfu.ca/~mjbrydon/tutorials/BAinPy/10_multiple_regression.html

[3] https://datatofish.com/multiple-linear-regression-python/

[4] https://machinelearningmastery.com/regression-metrics-for-machine-learning/

[5] https://www.geeksforgeeks.org/metrics-for-machine-learning-model/