Activation Function
Activate function is an vital element for designing a neural network. Choosing the activation function will give you entire control over the network model’s training procedure.
An activation function signifies the transformation of the input with the help of a weighted sum to the output. It utilizes a single node or further for the network to create the forecasting. You can also relate to the activation function as a transfer function. Or, you can call this a squashing function when the output range has limits. Activation functions can also be nonlinear and applied to as the non linearity of the network design.
Rectified Linear Activation Unit
ReLU stands for rectified linear activation unit and is viewed one of the many landmarks in the deep learning revolution. It's plain yet genuinely better than its precursor activation functions like as sigmoid or tanh.
relu activation function formula
Now how does ReLU make over its input? It uses this simple formula
f (x) = max (0, x).
ReLU function is its derivative both are monotonic. The function returns 0 if it receives any negative input, but for any positive value x, it returns that value back.
The function must also give further sensitiveness to the activation sum input and avoid simple contrast.
The result is to use the rectified linear activation function, or ReLU for little.
A node or unit that implements this activation function is applied to as a rectified direct activation unit, or ReLU for short. Frequently, networks that apply the rectifier function for the retired layers are related to as rectified networks.
Adoption of ReLU may effortlessly be accounted one of the many mileposts in the deep learning revolution, e.g. the approaches that now allow the routine development of veritably deep neural networks.
Gradient value of the ReLU function
In the deal of data for mining and processing, when we try to calculate the derivative of the ReLU function, for values lower than zero i.e. negative values, the grade introduce is 0. Which implicates the weight and the aptitudes for the learning function isn't streamlined consequently. This may lead to problems for the training of the model.
Summary :
In this blog, we learned about the activation function , ReLU function and gradient value of ReLU function. Check out more about tuples in Python here.