Different Activation Functions for Deep Neural Networks You Should Know

A quick snapshot of new and popular activation functions used in deep neural networks

Understand popular activation functions used in deep neural networks: Sigmoid, Softmax, tanh, ReLU, Softplus, PReLU, ReLU6, ELU, SELU, Swish, and Mish

Working of a Deep Neural Network

A deep neural network performs a linear transformation(z) using the node's input value(x), weight(w), and bias(b).

During forward-propagation, an activation function transforms the node’s linearity to non-linearity and decides if a neuron will be fired or not.

The output of the neuron is based on the strength of the signal that the activation function outputs. The activation function also normalizes the neuron's output to a value between 0 and 1 or between -1 and 1, depending on the activation function used.

The output layer calculates the error function, which is the difference between the predicted value and the ground truth.

The error calculated at the final output layer is then backpropagated across the neural network and the gradients using the activation function to update the weights and the biases. This helps reduce the…