Types of Activation Functions in Neural Network

vivekpandian
Nov 5 · 2 min read

What is an Activation Function?

The activation function is usually an abstraction representing the rate of action potential firing in the cell. In its simplest form, this function is binary — that is, either the neuron is firing or not.

What is the role of activation function in neural network?

The goal of the activation function is to introduce non-linearity into the output of a neuron.

Why do we need Non-linear activation functions ?

If you were to use linear activation functions (identity activation functions, eg: y=ax), then the neural network is just outputting a linear function of the input. In other words, no matter how many layers the network has, it will behave just like a single-layer perceptron, because summing these layers would give you just another linear function.

Types of Activation Function:

There are many types of activation functions. In this article, we are going to see the functions that I used in my projects along with python implementation for each function.

  1. Sigmoid
  2. Tanh
  3. Relu
  4. Selu
  5. Softplus
  6. Softsign

Sigmoid Function:

We are familiar with this function as we have used this in logistic regression.

Mathematical Equation : f(x)= 1/(1+e^(-x))

The value range is from 0 to 1.

Python Implementation: Kindly refer the Github link “https://github.com/vivekpandian08/activation-function

Tanh Function:

Tangent Hyperbolic function. It gives better results than using sigmoidal function but not the best.

Mathematical Equation : f(x)= (2/(1 + e-2x))-1

The value range is from -1 to 1.

Relu Function:

Rectified linear unit. It is the most used activation function in the hidden units for its not linear nature.

Mathematical Equation : f(x)= max(0,x)

It gives value of x when x is positive, or 0 otherwise.

The value range is from 0 to infinity.

Selu Function:

Scaled Exponential Linear Unit. It enables deep neural networks since there is no problem with vanishing gradients.

Mathematical Equation : f(x) = λ*x for all x>0 , λ* αe(x)−α otherwise.

The value range is from -infinity to infinity.

Softplus Function:

The research paper “Improving deep neural networks using softplus units” by Zheng et al. (2015) suggests that softplus provides more stabilization and performance to deep neural networks than ReLU function.

Mathematical Equation : f(x) =log(exp(x)+1)

The value range is from -infinity to infinity.

Softsign Function:

Softsign is an alternative to hyperbolic tangent activation function for neural networks. Even though tanh and softsign functions are closely related, the important difference is that tanh converges exponentially whereas softsign converges polynomially.

Mathematical Equation : f(x)=x/abs(x)+1

The value range is from -1 to +1.

In the next article, we will cover other important advanced activation functions such as reaky relu, parametric relu etc,.

Git hub Link : https://github.com/vivekpandian08/activation-function

Thanks for reading.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade