Shorts: How to choose an activation function?

Photo by Dan Cristian Pădureț on Unsplash

How to choose an activation function?

Activation function:

It is a black box that has some sort of rules and on that basis, it decides whether the output should be passed or not. The purpose of the activation function is to add non-linearity to the network.

As its name suggests, should the next neuron activate or not? that’s how it is named as an activation function.

Now the question is, how to select the activation function for our model?

The answer is simple and logical.

1. If you are doing Regression analysis, then the need is to understand the points which are nearer or far away from the regression line. If we see in our activation function zoo, then we should use the Linear activation function.

Activation functions [Source]

1. If you are doing binary class analysis for a classification problem then we can use a sigmoid or logistic activation function
2. If we are working on a multiclass classification problem, then we can use the softmax activation function
3. If we are working on a multi-label problem then we should use the sigmoidal activation function.
4. On top of that, if we are working on a convolutional neural network, then ReLu is the most preferable activation function

At last,

6. If we are working on the RNN [Recurrent neural network] problem, then we have good options like Tanh or Sigmoid activation function.

The graphs and formulas mentioned with each case can simply interpret why that activation is most suitable for that particular case.

These are the common scenarios whereby thumb rule, we can pick activation functions like this. Follow me for more interesting stories regarding data science, statistics, and computer vision.