Fundamental Machine Learning Concepts

Linear Regression

Linear Regression is one of the most fundamental Machine Learning algorithms. It is used in applications such as predicting house prices and played a central role in the development of neural networks.

To better understand Linear Regression, let's take the example of house prices.

Say you have 3 features to predict house prices: 

-X1: property size

-X2: number of bedrooms

-X3: number of bathrooms

We now want to use these 3 features to predict the house price. We can then set up the equation as the following:

Y is simply the house price and the X are just the input features. The β are the parameters multiplying the features. The value a is simply the y intercept.

Let's assume that these are the following parameters for predicting the house price:

β1: 20000

β2: 1000

β3: 1000

a: 30000

And now let's assume that our house's property size is 200 square meters, has 5 bedrooms and 5 bathrooms. Let's substitute these values into the equation.

Assuming these values, the linear regression model predicts that a house that has a property size of 200 meters squared, with 5 bedrooms and bathrooms costs 4,040,000$. This is of course not a perfect prediction but this is a simple demonstration of how a linear regression model works.

Linear regression theoretically doesn't have a limited number of features. Meaning we could also add other features such as number of kitchens, number of pools, etc. and keep going.

However, if the parameters are badly adjusted, the model's performance would sharply drop. But to determine the model's performance, there must be a way to measure it. 

Cost Function

Cost functions are used to measure "how badly" a model performs at a certain task. For example, if the model is terrible at predicting house prices the cost function will have a very large value, if it is good at predicting house prices, it will have a very low value. There are several different forms of cost functions, but for linear regression it is the mean squared error.

Continuing on the house prices from earlier, this is what the cost function would be

Gradient DEScent

This is the algorithm used to train most Machine Learning models. Gradient Descent is used to adjust the parameters of a model to improve its performance on and decreasing the output of the cost function.

To understand this algorithm in detail, we need to go back to our cost function. Last time, we used three parameters, let's just consider one "a" parameter for now. By changing this parameter, we also change the output of the cost function. This makes the cost function a function of "a", or f(a). 

We plot the values of f(a) for every "a" on a graph:

As we can see, different values of "a" lead to different outputs of f(a), the cost function. Our goal is to pick the "a" that has the lowest cost value. But when we randomly initialize the parameters, we usually never get the lowest value. So is there a way to adjust our value after our bad randomization?

That's where gradient descent comes in:

What gradient descent does here is that it changes the values in steps as denoted by the red arrows in the graph above. It iteratively updates the value, by taking steps, until it reaches the minimum value. 

But how does the algorithm decide in which direction to take these steps and what determines the size of their steps?

This is the equation that decides these things:

Let's break down what each component of the algorithm actually represents. The parameter alpha is a number between 0 and 1. This is the parameter that determines how large the step is in gradient descent, with larger values resulting in larger step sizes. 

The term right next to it, is just the derivative of the cost function f(a), in order words the instantaneous slope at that point. The direction of the step will simply be in the direction that yields a negative slope. On the graph, if the initial value of a is on the right side, it will move left, yielding a negative slope and on the left side, it will move to the right. 

A helpful analogy to consider is to imagine the point as a ball, and the graph being the landscape in which the ball rolls on. The ball will of course roll downwards if it is placed on a slope until reaching a valley, just like how gradient descent moves down towards a minimum.

By multiplying the alpha parameter with the derivative, we can thus adjust the size of the step and determine which the direction of that step. This term is subtracted from the previous value, ensuring that the direction of the term always points in the direction towards the minimum. If you try a positive value of a in f(a), the term will point to the negative direction and vice versa for a negative value of a. 

Since the alpha parameter adjusts the size of steps, the choice of its value will of course be important in determining how fast the algorithm converges to a solution. 

If we set the learning too low, like 0.0001, we will converge to the minimum value, it's just that it will take a very long time. If we set it too large like 0.1 or 0.2, some unexpected divergent behavior could occur and in the worst case, it could even increase the loss value instead of decreasing it. So the chosen alpha value has to be just right, not too large or too small, leading a fast but stable convergence.

You may ask yourselves, why not just use simple optimisation in calculus to always get the minimum? In the real world of machine learning, you might have millions or billions of parameters to adjust which we cannot visualize with a graph. The process of gradient descent works the same even in such high dimensions, pointing the value towards a minimum and slowly decreasing it step by step.

The example below shows an example cost function with two parameters. 

If we consider the parameters of the function to be "a" and "b". The update rule will be as follows:

As we can see, the update rules are very similar to our first example. One important thing to note however is that both updates get applied at the same time, which still points towards the direction that yields a negative slope until hitting a valley. 

If you understood this algorithm, then you would have a general high level grasp on how real world machine learning training works. The gist of it is pretty much that it adjusts the parameters in a way that decreases the cost function until hitting a minimum.

For the curious section (Advanced math)

Vectorization

If you wish to understand the actual mathematics used in AI, you need to understand vectorization. It is preferable that you have prior knowledge in linear algebra, but you should still be able to understand this section without any prior knowledge.

So why do we need to vectorize expressions? In real world applications, the number of weights and input variables can grow up to thousands of terms and in deep learning, these could be up to millions and billions of terms. Vectorization allows us to efficiently compute each term simultaneously, meaning that we do not need to consecutively or iteratively sum up each expression which would take way too long for that number of parameters. In order to vectorize these expressions, we need to transform these expressions from linear equations to vectors.

We are actually able to put the three β coefficients and input x variables into a vectors like so:

Since these two are vectors now, we can perform something known as a dot product operation, which will look like so: 

This actually gives the exact expression as in our linear regression equation from earlier:

The only thing that is missing from the expression is the bias term, which we can simply add. So finally the vectorized expression for the linear regression equation is:

This expression is much cleaner and actually runs much faster in code as well. This means that b and x could contain thousands of parameters and the expression would still work and hold true.