The Sigmoid and its Derivative

The simoid function, σ(x), is also called the logistic function, or expit [1]. It is the inverse of the logit function. Its function definition is:

Formula 1: The sigmoid function.

Let’s get familiar by plotting the function as has been done in the image above. The blue plot is the sigmoid function. And just to be complete, here is the python code that produced the above sigmoid plot:

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(1,1)
x=np.arange(-10,10,0.1)
y=1/(1+np.exp(-x))
ax.plot(x,y,linewidth=2)
ax.set_title("sigmoid function", fontsize=20)
ax.tick_params(axis="x", labelsize=15)
ax.tick_params(axis="y", labelsize=15)
#xs = np.arange(-10, 10, 0.1)
xs = np.linspace(1, 20, 10)
ys = np.linspace(0, 1, 1)
ax.hlines(y=0.5, xmin=-10, xmax=len(xs), colors='gray', linestyles='--', lw=1, alpha=0.7)
ax.vlines(x=0, ymin=0, ymax=len(ys), colors='gray', linestyles='--', lw=1, alpha=0.7)

plt.show()
fig.savefig("sigmoid", dpi=200, transparent=True)

We see that the sigmoid function maps from the real numbers to the [0,1] interval which can be interpreted as a probability. This and the fact that it is differentiable makes the sigmoid very suitable to be used with neural networks. Our goal now is to compute the derivative and write it in terms of σ(x) — you will see what I mean by that in a bit. So here we go. First we bring the formula into a slightly different form which is more suitable for doing the derivative:

Bringing the sigmoid function into a slightly different form by expanding the fraction by e^x.

Now we do the derivative:

In line 3 we apply the chain rule. In line 5 we substitute σ(x) = e / (e^x +1), i.e. formula 1 from above, and in line 6 we factor out σ(x). And this is it. This is the derivative of the sigmoid function in terms of itself, i.e. σ(x). We can see that the derivative of the sigmoid is simply itself multiplied by 1 minus itself.

To finish this up, we plot its derivative in the above figure in purple, so we can see the plot of the sigmoid and its derivative in the same image.

References
[1] https://en.wikipedia.org/wiki/Logistic_function.