Deep Learning Fundamentals Part 3
What are activation functions in Neural Networks ?
An activation function defines the output of the neuron given a set of inputs. As we saw in Part 2, we take the weighted sum of outputs from each neuron in the previous layer and pass that through the activation function which then transforms the input to a value between a lower and upper limit.
So what's up with this transformation and why is it needed ?
To understand let's look at an activation function called 'sigmoid'.

If the input to the sigmoid function is a large negative number it outputs a number close to zero. If the input is a larger positive number it outputs a number close to one. This output represents the amount of activation of the neuron.
Let's look at another activation function called 'relu' (in fact it is the most popular one these days) abbreviated from rectified linear unit.

Any input less than or equal to zero gets transformed to zero and if the input is greater than zero its gets passed to the output without any transformation.
Main idea which we can take from both these activation functions is that the more positive (or closer to the upper-limit) the output is, the more activated is the neuron by that specific input.
Below I have shown two different approach to specify activation function in Keras.
