Underfitting Effect

According to [4], supervised algorithms are said to have underfitting when a model is too simple to capture data complexities. It represents the inability of the model to learn the training data effectively resulting in poor performance both on the training and testing data. In simple terms, an underfit model is inaccurate, especially when applied to new, unseen examples. The reasons for Underfitting are:

• The model is too simple, so it may be not capable of representing the complexities in the data. 

• The input features which is used to train the model is not adequate representations of underlying factors influencing the target variable. 

• The size of the training dataset used is not enough. Excessive regularization is used to prevent overfitting, which constrains the model to capture the data well. 

• Features are not scaled. 

Techniques to Reduce Underfitting are: 

• Increase model complexity. 

• Increase the number of features, performing feature engineering. 

• Remove noise from the data. 

• Increase the number of epochs or increase the duration of training to get better results.

Overfitting Effect

From [4], supervised algorithms are said to be overfitted when the model does not make accurate predictions on testing data. When a model gets trained with so much data, it starts learning from the noise and inaccurate data entries in our data set. When testing with test data results in High variance.

In a nutshell, Overfitting is a problem where the evaluation of machine learning algorithms on training data is different from unseen data. Reasons for Overfitting are: 

• High variance and low bias. 

• The model is too complex. 

• The size of the training data. 

Techniques to Reduce Overfitting are: 

• Increase training data. 

• Reduce model complexity. 

• Early stopping during the training phase (have an eye over the loss over the training period as soon as loss begins to increase stop training). 

• For linear regression employ Ridge Regularization and Lasso Regularization. 

• Use dropout for neural networks to tackle overfitting.

The next figure extracted from [3] summarizes in visual terms both effects and the right adjustment of the models (in the middle). 

What is a Train-Test Split?

Setting a Train-Test Split

A numerical example dataset

import pandas as pd

import matplotlib.pyplot as plt

# scatter plot, dots colored by class value

df = pd.DataFrame(dict(x=X[:,0], y=X[:,1], label=y))

colors = {0:'red', 1:'blue', 2:'green'}

fig, ax = plt.subplots()

grouped = df.groupby('label')

for key, group in grouped:

    group.plot(ax=ax, kind='scatter', x='x', y='y', label=key, color=colors[key])

plt.show()

A numerical example of a Train-Test Split

A numerical example of a Train-Test Split for Linear Regression

Before creating a linear regression model using the training dataset, verify which input variables have a higher correlation with the output variable.

import seaborn as sns

import pandas as pd

import numpy as np

# Concatenate X_train and y_train

concatenated_data = np.column_stack((X_train, y_train))

# Create a DataFrame from the concatenated data

concatenated_df = pd.DataFrame(concatenated_data)

# Now, you can use sns.pairplot on concatenated_df

sns.pairplot(concatenated_df, diag_kind='kde')

# Display the pairplot

plt.show()

Now, let's create a multiple linear regression model.

# fit the model

#from sklearn.ensemble import RandomForestRegressor

#model = RandomForestRegressor(random_state=1)

from sklearn.linear_model import LinearRegression

model = LinearRegression()

model.fit(X_train, y_train)

Another possibility is to use statsmodels library to extract model parameters.

# with statsmodels

import statsmodels.api as sm

xm = sm.add_constant(X_train) # adding a constant

model2 = sm.OLS(y_train, xm).fit()

print_model = model2.summary()

print(print_model)

Logistic regression vs. linear regression

The main difference between logistic and linear regression is that logistic regression provides a constant output, while linear regression provides a continuous output [6].

In logistic regression, the outcome, or dependent variable, has only two possible values. However, in linear regression, the outcome is continuous, which means that it can have any one of an infinite number of possible values. Logistic regression is used when the response variable is categorical, such as yes/no, true/false, and pass/fail. Linear regression is used when the response variable is continuous, such as hours, height, and weight.

For example, given data on the time a student spent studying and that student's exam scores, logistic regression and linear regression can predict different things.

With logistic regression predictions, only specific values or categories are allowed. Therefore, logistic regression predicts whether the student passed or failed. Since linear regression predictions are continuous, such as numbers in a range, it can predict the student's test score on a scale of 0 to 100.

The next figure helps to illustrate a comparison between both regressions [7]. 

A numerical example of a Train-Test Split for Classification

We will demonstrate how to use the train-test split to evaluate a random forest algorithm on the sonar dataset. The sonar dataset is a standard machine learning dataset composed of 208 rows of data with 60 numerical input variables and a target variable with two class values, e.g. binary classification. The dataset involves predicting whether sonar returns indicate a rock or simulated mine:

Sonar Dataset (sonar.csv),

Sonar Dataset Description (sonar.names).

The example below downloads the dataset and summarizes its shape.

Finally, it is possible to employ code to perform the Logistic regression.

# fit the model

#model = RandomForestClassifier(random_state=1)

model = LogisticRegression(random_state=1)

model.fit(X_train, y_train)