Most Used Activation Functions In Deep Learning

Activation functions introduce non-linearity, allowing the network to learn complex patterns.

1. Sigmoid(Logistic) Activation Function:

• Commonly used in the output layer of binary problems.
• Range (0,1)

2. Hyperbolic Tangent (tanh):

• Used in hidden layers of neural networks.
• Range (-1,1)

3. Rectified Linear Unit (ReLU):

• One of the most used ones in the hidden layer.
• However, it may suffer from the dying ReLU problem, where neurons can become inactive and stop learning.
• Range [0,+∞)

4. Leaky ReLU:

• It addresses the dying ReLU problem by allowing a small, non-zero gradient for negative values.
• Range (-∞,+∞)

5. Parametric ReLU (PReLU):

• Similar to leaky ReLU, but with an alpha as a learnable parameter.
• Range (-∞,+∞)
• f(x) = max(ax, x)

6. Softmax Activation Function

• Typically used in the output layer of the multi-class classification problem.
• Range (0,1)

• k: number of classes in the multi-class classifier.