Pre-Lab

By the end of this learning module, students will be able to:

What is a Neural Network (NN)?
Elements of the NN model.
Why activation functions?
Terms of activation functions.

Why Neural Network?

Neural Networks are crucial in modern artificial intelligence (AI) and machine learning because they excel at identifying patterns and making predictions from large, complex datasets. Inspired by the structure and function of the human brain, neural networks consist of interconnected layers of nodes (Neurons) that process data and learn from it through training. Their ability to automatically learn and recognize patterns from data makes neural networks especially useful in domains where traditional methods can't handle complexity, leading to significant advancements in technology and industry.

Key elements of a Neural Network.

Neurons (Nodes): Basic units that process input data and pass it on to the next layer after applying an activation function.
input Layer: The first layer that receives the raw data.
Hidden Layers: These are called hidden because we can't control their flow or how the input goes into these layers.
Weights: Parameters that determine the importance of inputs to a neuron.
Activation Function: The function adds non-linearity to a neuron's output, allowing the model to learn and capture complex patterns.

Activation Functions

It known as transfer functions, turns the weighted inputs into a nonlinear output. This introduces nonlinearity into the network, allowing it to learn complex patterns. Without activation functions, a neural network would behave like a simple linear model, regardless of how many layers it has. The activation function decides whether a neuron should be activated or not by calculating the weighted sum and further adding bias to it. They also help limit the output to a specific range, making the model more effective.

Linear Activation function

In neural networks, a linear function is often used in the output layer when the task is regression. A linear function has an equation similar to that of a straight line i.e. y = mx

The linear activation function has a range of −infinity to +infinity and is typically used only in the output layer for regression tasks.
A key issue with linear functions is that if we try to introduce non-linearity by differentiating them, the result becomes constant, losing dependence on the input x.
For example: The calculation of the price of a house is a regression problem. House prices may have any big/small value, so we can apply linear activation at the output layer.

Sigmoid Function

The Sigmoid function is commonly used in neural networks, particularly in binary classification tasks, because it maps input values to a range between 0 and 1, making it ideal for predicting between two possible outcomes. It is a function that is plotted as an ‘S’ shaped graph.

Equation : Y = 1/(1 + e-x)
For input values X between -2 and 2, the output Y changes rapidly.
Use cases: The result can be predicted easily to be 1 if the value is greater than 0.5 and 0 otherwise.

Figure 1: Sigmoid function.

Tanh Function

The Tanh function is used in neural networks to activate neurons by scaling their outputs between -1 and 1. This helps the network learn faster and better by centering the activations around zero, This makes learning for the next layer much easier. It’s actually a mathematically shifted version of the sigmoid function

Equation : f(x) = tanh(x) = 2/(1 + e-2x) – 1

RELU Function

The ReLU (Rectified Linear Unit) function is used in neural networks to introduce non-linearity and speed up training. It outputs zero for negative values and keeps positive values unchanged, helping the network learn faster and more effectively by avoiding issues like vanishing gradients. ReLu is less computationally expensive than tanh and sigmoid because it involves simpler mathematical operations. At a time only a few neurons are activated making the network sparse and making it efficient and easy for computation.

. Equation: relu(z) = max(0,z); outputs zero for any negative input and passes positive input through unchanged

GELU Function

GELU (Gaussian Error Linear Unit) helps in stabilizing the training of deep networks by maintaining a balance between linear and non-linear transformations, which can improve model performance. It combines the properties of ReLU and dropout, providing smoother gradients and a probabilistic approach to activation. Deep models like GPT, BERT, and most transformers use GELU.

Equation: 👇

where erf denotes the error function

Softmax Function

Softmax function is mainly used for multi-class classification tasks, particularly in the final layer of the network. It converts the raw output scores (logits) from the neural network into a probability distribution over the classes. Each score is exponentiated and normalized so that the sum of all probabilities equals 1.

Equation: 👇

Here, zi represents the input to the softmax function for class i, and the denominator is the sum of the exponentials of all the raw class scores in the output layer.

----------------------------------------------------------------------------------------------------------------------------------------------------------------------

Here we utilized the mail.csv dataset as a sample to evaluate the performance of different Neural Network activation functions and we got the following results.

Page updated

Google Sites

Report abuse