Types of Activation Functions in Neural Network

What is an Activation Function?

The activation function is usually an abstraction representing the rate of action potential firing in the cell. In its simplest form, this function is binary — that is, either the neuron is firing or not.

What is the role of activation function in neural network?

The goal of the activation function is to introduce non-linearity into the output of a neuron.

Why do we need Non-linear activation functions ?

If you were to use linear activation functions (identity activation functions, eg: y=ax), then the neural network is just outputting a linear function of the input. In other words, no matter how many layers the network has, it will behave just like a single-layer perceptron, because summing these layers would give you just another linear function.

Types of Activation Function:

There are many types of activation functions. In this article, we are going to see the functions that I used in my projects along with python implementation for each function.

1. Sigmoid
2. Tanh
3. Relu
4. Selu
5. Softplus
6. Softsign

Sigmoid Function:

We are familiar with this function as we have used this in logistic regression.

Mathematical Equation : f(x)= 1/(1+e^(-x))

The value range is from 0 to 1.

Python Implementation: Kindly refer the Github link “https://github.com/vivekpandian08/activation-function “

Tanh Function:

Tangent Hyperbolic function. It gives better results than using sigmoidal function but not the best.

Mathematical Equation : f(x)= (2/(1 + e-2x))-1

The value range is from -1 to 1.

Relu Function:

Rectified linear unit. It is the most used activation function in the hidden units for its not linear nature.

Mathematical Equation : f(x)= max(0,x)

It gives value of x when x is positive, or 0 otherwise.

The value range is from 0 to infinity.

Selu Function:

Scaled Exponential Linear Unit. It enables deep neural networks since there is no problem with vanishing gradients.

Mathematical Equation : f(x) = λ*x for all x>0 , λ* αe(x)−α otherwise.

The value range is from -infinity to infinity.

Softplus Function:

The research paper “Improving deep neural networks using softplus units” by Zheng et al. (2015) suggests that softplus provides more stabilization and performance to deep neural networks than ReLU function.

Mathematical Equation : f(x) =log(exp(x)+1)

The value range is from -infinity to infinity.

Softsign Function:

Softsign is an alternative to hyperbolic tangent activation function for neural networks. Even though tanh and softsign functions are closely related, the important difference is that tanh converges exponentially whereas softsign converges polynomially.

Mathematical Equation : f(x)=x/abs(x)+1

The value range is from -1 to +1.

In the next article, we will cover other important advanced activation functions such as reaky relu, parametric relu etc,.

Git hub Link : https://github.com/vivekpandian08/activation-function

Thanks for reading.