What is the sigmoid function?
A sigmoid function is a mathematical function having a characteristic “S”-shaped curve or sigmoid curve. It transforms any value in the domain (−∞,∞) to a number between 0 and 1.
Applications
The sigmoid function’s ability to transform any real number to one between 0 and 1 is very useful in Data Science and many other fields such as:
- In deep learning as a non-linear activation function within neurons in artifical neural networks to allow the network to learn non-linear relationships between the data
- In binary classification, also called logistic regression, where the sigmoid function is used to predict the probability of a binary variable.
Issues with the sigmoid function
Although the sigmoid function is very popular in the context of gradient descent the gradient of the sigmoid function is in some cases problematic. The gradient vanishes to zero for very low and very high input values and this makes it hard for some models to improve.
For example during backpropagation in deep learning the gradient of a sigmoid activation function is used to update the weights & biases of a neural network. If these gradients are very small the updates to the weights & biases are very small and the network will not learn.
