Understanding a Single Neuron’s role in Neural Network.

Most of you might have heard the term Neural network(if you didn’t, it’s okay) and tried to apply it to some standard problems, but you might have noticed, when it comes to implementing them, you need to take the support of many libraries, which on one hand makes its implementation easy but at the same time it takes away that practical intuition, “feeling”, of the neuron we need to have in order to be comfortable in applying it on any new problem. So in this article, we will walk through a simple intuition of how a neuron functions in python(without using many libraries).

Src: Sikich

So, before we move further, let’s see the abstract definition of the neural network which is used for practical purposes-

A Neural Network is a collection of neurons(which are analogous to the human brain), which are connected to every other neuron on the next layer(outgoing) and to the layer just before it(inputs). These connections have a value associated with them which is known as ‘weights’ and every neuron has a bias which helps in giving the right direction. Now after processing them, we apply an activation function that produces an output that predicts the most desired(according to input)result.

Source: Otexts

Now, let us get the start by one single neuron by considering its input scenario,i.e. what a single neuron’s input processing code might look like… We know that a single neuron will take input from all the neurons from the previous layer, multiply them by their weights and add a bias, so let’s pick up one of the ‘blue colored neuron’ and write about that.

NOTE: here we are not considering the whole network or the whole layer. We are just looking at what a single neuron is doing.

inputs=[1,2,3,4]          #from the green neurons
bias=2
weights=[0.2,0.5,0.9,0.1] #Corresponding to the four arrows(edges)
output=inputs[0]*weights[0]+inputs[1]*weights[1]+ inputs[2]*weights[2]+ inputs[3]*weights[3]+ bias
print(output)

This is same as the dot product as shown(if you don’t know what dot product is you can find an intuitive explanation here):

Dot Product

import numpy as np
inputs=[1,2,3,4]  #from the green neurons
bias=2
weights=[0.2,0.5,0.9,0.1] #Corresponding to the four arrows(edges)
output=np.dot(inputs,weights)+bias
print( output)

The output is: 6.300000000000001

This is a pretty simple code about how a single neuron processes its input. Now after it processes its input values, we have to apply an ‘Activation function’ to the obtained value to get the final output of the neuron.

What is Activation Function & Why we use it??

Before discussing various kinds of activation function, let us look into why we need an activation function-

So in simple terms, the basic task of the activation function is to provide a way to decide to activate(produce a significant value as output) a neuron based on its weights and bias.

Also, it helps in introducing non-linearity to the neural network’s output, and without it will be able to perform only linear mapping between x(inputs) and y(outputs).

Without an activation function, the output will be just the dot product of the input vector and the weights matrix, which is added with bias, since all the terms in the calculations are linear the output will also be a linear line.

Usually, the more complex the data is, from which we are trying to learn(make predictions), the more non-linear the mapping from the features to the ground truth label.

Types of activation functions

Now that we know why we need an activation function, it is the right step to look for some of the activation functions. There are a variety of activation functions available, let us look at the popular ones-

Sigmoid Activation-it’s a mathematical function that maps any value within the range of 0 to 1, as you can notice in the image the value never reaches exactly 1or 0 it just tends to be 1 or 0.

Source: https://www.researchgate.net/figure/An-illustration-of-the-signal-processing-in-a-sigmoid-function_fig2_239269767

TanH Activation- it’s a mathematical function, which is also called the Hyperbolic Tangent function, which maps the values between -1 and +1. Same as sigmoid it also never reaches the exact value of 1 or -1.

Source: https://www.experfy.com/blog/activation-functions-within-neural-networks

ReLU Activation-it stands for Rectified Linear Unit, it is the most common and popular activation function.it maps negative values to 0 and positive values to themselves, the image will make it more clear.

Source: https://kraj3.com.np/blog/2019/11/non-linear-activation-functions/

SoftMax Activation- this activation function is used only in the output layer of the neural network, only when we need probability scores during classification problems. In this function, the output value not only depends on its input but also on the input of the other neurons of the same layer, as this function gives the probability of that particular(particular to neuron) output. The sum of the output of all the neurons is always 1, as it gives the probability.

Source: https://stackoverflow.com/questions/58999818/constrain-activation-on-keras-neural-net-output-layer

Now that we know of various activation functions, let us apply the activation function using python on our sample data used above. Since the sigmoid function is the common one so let us apply it here.

import numpy as np 
import math
inputs=[1,2,3,4]  #from the green neurons
bias=2
weights=[0.2,0.5,0.9,0.1] #Corresponding to the four arrows(edges)
output=np.dot(inputs,weights)+bias
result= 1/(1 + np.exp(-output))
print( result)

The output is:0.9981670610575072

Similarly, you can apply other activation functions too.

Error

Now let us talk about the error in the output which tells us how much our current output deviates from the true output.

In our sample case let our true output be 1(we are considering only 1 neuron in our case, which has produced the above output.). So to calculate the error we do the summation of the squared difference of the obtained and the true value. This is also known as the loss function.

Error is denoted by J. y and y^ are the obtained and true value respectively.

So using this formula, the error in our case is calculated as

import numpy as np 
import math
inputs=[1,2,3,4]  #from the green neurons
bias=2
weights=[0.2,0.5,0.9,0.1] #Corresponding to the four arrows(edges)
output=np.dot(inputs,weights)+bias
print(output)
result= 1/(1 + np.exp(-output))
print( result)
error=1/2*((result-1)**2)
print(error)

The error comes out to be: 1.6798325834533343e-06

As we know that no one can predict the right amount of weights needed to get the error-free output. We usually initialize the weights with random values and then look at the error value at the output layer, which is generally large initially but we need a way to converge to the optimal weight value to get the model ready for prediction.

Now we should do something to reduce this error, to do this, we apply an algorithm called Backpropagation.

Backpropagation

It is an algorithm that looks for minimizing the error, in the weight space using a technique called gradient descent. If you do not know what gradient descent is, here is an informative explanation you can refer to

Backpropagation- the weights are updated based on the rate of change of error w.r.t. rate of change of weight.

The weights are updated by adding/subtracting the derivative(error w.r.t. weight) times(multiplied) the learning rate referred to as “ETA”. So using this we always make a move towards the more optimized weights vector, we stop when we get the error below a certain value(which is very small).

If you want to know about the calculus involved in Backpropagation, then Check out its explanation from the below site.

What Is Backpropagation? | Training A Neural Network | Edureka

I hope you found this article beneficial and now have a clearer view of what a neuron does. If you have any suggestions or questions, feel free to contact me by leaving a message.