Intro to Deep Learning

Neural NEtworks

Neural networks are the backbone of many of the newer architectures like Transformers that power LLMs or CNNs for image recognition tasks.

We will use a very simple neural network to understand how they work. 

The inputs to the neural network are simply the features we want to use to predict something. For this example let's consider that this neuron predicts wether an email is spam or not. The features in this case would be the number of suspicious key words and percentage of links.

Each neuron in the hidden layer is then responsible for using the features to understand their behavior. Imagine one neuron detecting a keyword pattern, another detecting the URL pattern and the last one detecting the sender behavior. The output neuron, using the patterns found in the hidden layer then makes a prediction, determine the percentage chance that the email is spam. 

Let's just consider one neuron for now and understand what it is actually doing:

The math behind this neuron is actually relatively simple, it is just:

Does this seem familiar? Because this is just the equation from linear regression. That's it, that is the equation for one neuron. The neuron uses this computation to detect and find patterns in the features. Now the hidden layer simply does this three times, as seen below:

Now we have three values produced by the neurons, a1, a2 and a3. Now if we just leave them like this, our neural network would just be an overly complicated linear regression model, not anything special. This means that our network won't be able to learn the complex patterns given by the features.

This is where activation functions come in. The activation functions help the neural networks turn on when something is important and off if otherwise. 

One of the most commonly used activation functions is ReLU.

After applying ReLU to a1, a2 and a3 these values are then passed to the final output layer. Without any other functions, our output layer will just be this again:

This however is just a linear output. For various applications, we would apply another function to the final linear output. For our case, since we are only trying to predict wether an email is spam or no spam, we will use a function that outputs a number between 0 or 1.

Luckily for us, the sigmoid function does just that

This network has two hidden layers in total, it should be noted that the input layer are not neurons, they are simply the inputs. The nodes in the hidden layer are neurons.

Forward pass works very similarly to the previous one layer example, except the inputs from the previous hidden layer have to be passed to the next hidden layer. 

You could intuitively understand backpropagation as simply calculating the gradients of the neural network by going backwards. Essentially you start at the output layer, then compute the gradients of the weights between the hidden layer and output layer. Using that gradient, you can then calculate the previous layer before that layer and repeat until you reach the beginning. Just like in linear regression, these gradients are then used to update the weights during gradient descent to find the most optimal configuration of every weight.

For the curious section (Advanced math)

This section requires some knowledge of the chain rule, derivatives and linear algebra operations like matrix multiplication, transposing and dimensions. We will use the same neural network as the one we used at the beginning:

Now we have to vectorize the forward pass equations. The answer to this computations, the hidden layer values, can be expressed as a vector, we also notice that the b terms can also be comfortably expressed as a vector, the challenge now lies in finding an expression for w and x

Given what we know so far, we can express everything into this for now:

To express W, we can't just use vectors, we actually have to use matrices. The dimension of vector a is (3,1). The two dimensions of x is (2,1) and b is also (3,1). 

If W is a (3,2) matrix and it is multiplied with x (2,1), the answer will be (3,1). Knowing this, we can now fill in the question mark part and see the final result:

As we can see, each component of the resulting vector is directly equal to the expression we saw in the diagram 1:

Hence we can comfortably express the equation for the first layer as:

Using this knowledge, we can fully construct all the equations outlining the forward pass in this neural network, remember z is the final value of the node(s) in the neural network after activation functions are applied:

The section below will contain math from multivariable calculus and linear algebra. If you have pre requisite knowledge of these topics you can continue. But do note you do not need to understand the math here to be able to write pytorch code. The most important math you should understand is the math above

Before we find the expressions to calculate the gradients, we must establish a few things very clearly.

First we must understand the dimensions for all the components:

x: (2,1)

W1: (3,2)

b1: (3,1)

a1: (3,1)

z1: (3,1)

W2: (3,1)

b2: (1,1)

a2: (1,1)

z2: (1,1)

Each expression or derivative for each corresponding weight must have the same dimensions, so if W1 has a dimension of (3,2), it's gradient also has to have a dimension of (3,2). We also need to know that when we find the gradients, it is the gradient of the cost function with respect to a variable.

This means the gradient used to update W1 is dL/dW1. We can't directly find dW1 however, because it is nested between many layers of functions, which is why the chain rule needs to be involved in this process. 

We can begin our derivation process by assuming that we know the gradient of the cost function with respect to z2, because we have to start at the very end of the neural network during backpropagation.

First we need to compute dz2/dza2. We will be able to find the expression for dL/da2, which will allow us to find the gradients for W2, b2 and z1. The dimension for this expression is again (1,1)

Now we can continue onwards to find W2, b2 and z1.

To make this process easier, let's just set dz1 to some variable like we did with dz2:

From dz1, we can finally calculate a1:

Then we can finally find W1 and b1:

The gradients we actually need are from W1,b1,W2 and b2, as they are the actual weights we need to update. So here is a quick summary of the gradients we actually need:

While the derivation process was very complicated, the final answer is actually not and there even seems to be a pattern to it. Each b term gradient's expression seems identical to all other layers while the W terms are just the b terms but multiplied by the transpose of their inputs.

So congratulations if you have even decided to come down this far, it's ok not to understand all the maths here but if you understand the general idea of backpropagation, which is going back to each layer and iteratively computing the gradients, then that is already pretty good.