A Simple Neural Network With A Single Neuron
Hello guys,
It’s Nishank here, and today we will learn, how to make a simple neural net and predict some numeric output using that. A neural network is a simple mathematical unit, which takes some input and predicts something on the basis of training it received earlier.
Show your support by subscribing to our newsletter!
A neural network is made up of many neurons which help in computation. A single neuron has something called a weight attached to it, also called synaptic weight. These weights changes in the direction of our prediction when we train our neural network.
So the focus of this post is, creating a neural network with a single neuron, training it for 10000 runs, predicting the output in every run, obtaining the error by comparing it with our expected output, adjusting weight based on the error and then finally try to predict the actual output.
The best part about this is, we will not be using any libraries for making our neural net (except numpy obviously). This would help us in understanding the basic structure of a neural network and how it actually works.
Without making it anymore overwhelming for you, let’s start with the code. We will start by installing the library required, in this case numpy .
Then we will import the libraries required. We will need exp for exponential (you will discover why, later), array (for array obviously), random for generating random numbers and dot for doing the dot product (right the physics one!) all from numpy.
Okay, so we will be using the following function for our program training and testing purposes as data.
You must be wondering how the heck did he got that output? Don’t you worry, no rocket science here, I just copy-pasted the entire Input 2 row of the table!
So we will be using this data set. Now, moving on, let’s declare the input and output variables for our program, which would both be arrays that represent this table.
The .T used in train_outputs is used to transform the array, thereby making it a vector so that it can be multiplied easily and so that we can . tally it easily with the table . Also, train_inputs is an array of arrays, which will be used in our model. Notice, that I have exactly replicated the table above in our input and output variables.
Our next step in generation of random synaptic weights for our neuron at the starting of training. These generated weights will be adjusted as our program progresses. We will do so by using random function.
We have used seed() function here to keep the generated random weights the same, each time the program is executed, until it is closed.
Now here is a concept worth understanding. Well it’s more of an application than a concept. We need to generate random numbers for a neuron with 3 input connection and 1 output connection, so our function becomes random(3,1).
Also we need to generate random numbers in the range -1 to 1 (since our output is either 0 or 1). While using the random function for range a to b , the random function is defined as : (b — a) * random_sample() + a , so here a = -1 and b = 1 and the function becomes something like this, where synaptic_weights = 2 * random.random((3, 1)) — 1.
Next, we will create the main part of our neuron, i.e the function we will use to train it. Because a neuron without training is as good as a piece of log.
Don’t be intimidated by this simple function (yeah simple!), let’s break it into small pieces. the train function contains 4 main blocks : the head, output predictor, error calculator and weight adjuster. It’s that simple!
The Head
The first part of train() function is the head. It accepts 3 arguments, namely train_inputs (the inputs for training), train_outputs (the outputs for training) and iterations (the number of time the loop will run to train the neuron, so that it’s weights could be adjusted.).
There is also a for loop which runs for the number of iterations provided. Here xrange is used to specify the number of time the loop should run. The remaining 3 parts of train() function are inside this loop.
Output Predictor
This part of train() is used to predict output for the given train_inputs using the current synaptic_weights. This is done by using getoutput() function, which can be defined as follows:
Here we will be using sigmoid value of inputs and synaptic_weights. For combining these, we have used dot() function to perform the dot product of inputs and synaptic_weights and sending them into a sigmoid() function.
A sigmoid function is a common function used in neural networks to predict the output. What it does, is it normalizes the value of the dot product between 0 and 1(exactly the thing with need!).
It looks something like this:
Let’s put this concept into code:
Error Calculator
The next part is calculating the error. It is simple step that subtracts the output we got in Output Predictor part from the actual expected output stored in the train_outputs variable.
Weight Adjuster
The last and the most important part of train() function. Here synaptic_weights are adjusted using the error obtained in Error Calculator.
The sigmoid curve looks something like this -
Notice, that the line gets straighter and constant as we move towards the edges. That means the gradient increases and the surety of getting the right prediction increases. We will also use this concept of sigmoid gradient in our program to get more accurate values.
Sigmoid Gradient = output * (1-output)
Here, first we have given our output from Output Predictor to sig_grad() function. What it does is, it calculates the Sigmoid Gradient of the output using the formula above so that we can move towards the edges of the sigmoid curve and get more accurate results.
Next, we will multiply our error value to sigmoid gradient of output to modify our outputs accordingly and finally, we take the dot product of these and the transpose (.T) of train_inputs to get our adjustment.
The final step of each run and train() function is to update the synaptic_weights by adding adjustment to it. We declare the variable as global synaptic_weights to access the global variable called synaptic_weights (because in python, definition and declaration are together and it would give an error if we update synaptic_weights before declaring them, also they would be present in local scope).
The rest of the program is just passing the values into train() function and then printing the results.
We just printed the Random Starting Synaptic Weights, and the called train() function by passing train_inputs , train_outputs and 10000 as our number of runs.
Next we printed the synaptic weights after training and the tested the neural net with a custom input of [1,1,0] , ideally we should get 1 as it is in the 2nd row and we just copied the 2nd row in the output (see the top of this post). Let’s see what happens!
Observe that our output is [0.9999225] , which is very close to 1 , exactly the output we wanted. Also the synaptic weights have changed and look much more consistent. We trained our model 10000 times and it can predict output so closely, and this is just one neuron, imagine the power of neural nets with 1000s of neurons!
I hope you enjoyed reading this post and it was both informative and interesting and it improved your practical understanding of neural nets. You can get the entire code of this post by clicking the banner below. Don’t forget to subscribe!
