Let’s activate your activation(function) in Deep learning
Deep learning is a very nice approach to solve automation problems in this age of technology.The data generated everyday is huge volume.To use this data correctly we can solve many great problems using machine learning and deep learning.
Deep learning mainly uses artificial neural network to build models and solve problems.First we need to know:
What is artificial Neuron??
An artificial neuron is a connection point in an artificial neural network. Artificial neural networks, like the human body’s biological neural network, have a layered architecture and each network node (connection point) has the capability to process input and forward output to other nodes in the network. In both artificial and biological architectures, the nodes are called neurons and the connections are characterized by synaptic weights, which represent the significance of
the connection. As new data is received and processed, the synaptic weights change and this is how learning occurs.
Now here we introduce a new concept which is an Activation function help to define the output of a node given an input or set of inputs. A standard computer chip circuit can be seen as a digital network of activation functions that can be “ON” (1) or “OFF” (0), depending on input.
ACtivation function is like an electronic switch to send response to another neuron.There is many types of activation function.We discuss about them which you have to choose for your need in the deep learning problem with visualisation.
A model heavily reliable on the activation function.If you apply linear function on a non linear problem,your model will be very poor and low accurate.
The types of Activation function mainly two categories:
1.Linear function
2.Non linear function
Here we discuss about various activation functions Likerelu,sigmoid,Linear,Tanh.
- Relu(Rectified Linear Activation Function:
The rectified linear activation function is a piecewise linear function that will output the input directly if is positive, otherwise, it will output zero. It has become the default activation function for many types of neural networks because a model that uses it is easier to train and often achieves better performance.
- Sigmoid:
It is also a nonlinear function with a S shaped curve which called sigmoid curve and function is called sigmoid function.
A standard choice for a sigmoid function is the logistic function shown in the first figure and defined by the formula.
It is a bounded, differentiable, real function that is defined for all real input values and has a non-negative derivative at each point.A sigmoid “function” and a sigmoid “curve” refer to the same object.
- Bipolar:
The Bipolar activation function used to convert the activation level of a unit (neuron) into an output signal. It is also known as transfer function or squashing function due to the capability to squeeze the amplitude range of output signal to some finite value .
- Hyperbolic Tangent(TanH):
The hyperbolic tangent function is another type of AF used in DL and it has some variants used in DL applications. The hyperbolic tangent function known as tanh function, is a smoother [1] zero-centred function whose range lies between -1 to 1.
- Exponential Linear Units, ELU
Exponential Linear Unit or its widely known name ELU is a function that tend to converge cost to zero faster and produce more accurate results. … They are both in identity function form for non-negative inputs. On the other hand, ELU becomes smooth slowly until its output equal to -α whereas RELU sharply smoothes.
- SoftPlus
softplus is a newer function than sigmoid and tanh. It is firstly introduced in 2001. Softplus is an alternative of traditional functions because it is differentiable and its derivative is easy to demonstrate.
- Piecewise Linear
In mathematics and statistics, a piecewise linear, PL or segmented function is a real-valued function defined on the real numbers or a segment thereof, whose graph is composed of straight-line sections. It is a piecewise-defined function, each of whose pieces is an affine function.
Total code for this is below
Conclusion:
So Neural network activation functions are a crucial component of deep learning. Activation functions determine the output of a deep learning model, its accuracy, and also the computational efficiency of training a model — which can make or break a large scale neural network. Activation functions also have a major effect on the neural network’s ability to converge and the convergence speed, or in some cases, activation functions might prevent neural networks from converging in the first place.
So CHOOSE WISELY YOUR ACTIVATION:
source i used:
