The Joy of AI: let’s build a perceptron part 1
Introduction to p5.js
This is a basic introduction to Artificial Intelligence or Machine Learning using a coding language called p5.js and a machine learning library called ml5.js. This can all be done in your browser for free, nothing to download, nothing to pay or subscribe to. If you have never coded before or are unfamiliar with p5.js then I recommend you download a free ebook (PDF) from my website at www.elegant.ai, or click on the link HERE.
What is a perceptron
A perceptron is the most basic part of a neural network, something that underpins most artificial intelligence and machine learning. So we are going to start at the most basic level. This is an attempt to create one rather than just trying to explain it in a theoretical way. The best way to learn is by doing in my experience as an educator. If you think of a human brain as the neural network, the perceptron is a single neuron.
Building a Perceptron
We are going to build a very simple perceptron where we have two inputs (x, y) which are the co-ordinates of lots and lots of small circles. We then draw a line across the canvas and all those circles above the line are filled in black and all those below the line are filled in white. This is such a simple thing that you could do this with a ruler and coloured pencils.
Yet this is a great opportunity to choose something trivial to demonstrate how a perceptron can be trained to do this. We have to give it the correct data for each circle and allow it to tweak itself so that it can, slowly over time, learn to get it right. But first let us have a look at what we need for a perceptron.
Weights
The above diagram is very simple but it is missing a number of things, firstly each input (x and y co-ordinate) is multiplied by what is called a ‘weight’. This is a random number between -1 and 1. The reason for having these weights is for one very good reason it uses them to improve its guess.
Let us take one circle with co-ordinates x = 10, y = 350. The perceptron does not know whether it is above or below the line (wherever we draw the line) so it gives each input another random value to a variable called weight. We will call them weightX and weightY. We will give weightX = 0.4 and weightY = -0.9.
We then multiply the x value by the weightX which is 10 x 0.4 = 4, followed by y times weightY which is 350 x -0.9 = -315. Then we add them together -315 + 4 = -311. This gives us a negative number which means it must be below the line. Yet we haven’t even drawn a line yet so how does it know? Well, it has made a guess (a wild guess).
The value of -311 is passed through what is called an activation function which is inside the perceptron. We will look at the one we are using in the next section.
Activation Functions
Activation functions are very important to neural networks and in our very simple example we are going to use a very simple one called a step activation function. The perceptron is going to give an output based on the input it has received (the inputs times the weights added together to give us the final value of -307).
4 + (-311) = 307
If it receives a negative value it returns -1 and if it receives a positive value it returns +1. This is how it decides if the circle is above or below and whether it is going to be a black circle or a white circle. Below is a step activation function where the output (y axis) is either -1 or +1 no matter how big the positive or negative input value is.
Bias
We have a slight problem though, what happens if the input is zero. Is it +1 or -1. We could just hard code it in and make a decision one way or the other. Instead we introduce a bias. This means that there is always an input of some value just incase we have drawn our circle at (0, 0). This bias has an input of 1 and has its own random weight. We now have a third calculation. We will call the input Bias and the weight for that bias weightB. We multiply Bias times weightB where Bias has a fixed value of 1 and weightB has a random value (say 0.6).
The calculations are now:
10 x 0.4 = 4 (from the x input)
350 x -0.9 = -311 (from the y input)
1 x 0.6 = 0.6 (from the bias)
The sum of the inputs is now 4 – 311 + 0.6 = -306.4
You can see how one input (350) has a more dramatic effect than the bias (1) or the other input 10. So we would normally normalise the inputs the negate the impact of of very large to very small inputs.
Normalisation
Normalisation is to take all the inputs and scale them to values between 0 and 1. We can do this by dividing by 400 which is the width and height of our canvas. So the inputX becomes 10 divided by 400 = 0.025 and the inputY becomes 350 divided by 400 = 0.875. The values have been scaled (or mapped) to fit in the range of zero to one.
Neural networks work best with values between 0 and 1 or -1 and +1. So there is a need to normalise the data, with the dimensions of the canvas being easy to calculate in this example but ML5.js and other machine learning libraries do it for you with built in functions.
Feedforward
What we are doing is feeding forward the data. We have the inputs, then multiply by their weights, add them all together and push them through the activation function and get an output of either +1 or -1. This is the essence of the perceptron and the process is called feedforward. The weights are tweaked and the inputs fed through again, this is repeated many times.
Your question might be how does it know what to tweak the weights. This is because the data we give it has the answer based on where the circle is in relation to the line. It makes a guess, it checks its guess with the desired (target) answer and gets an error in return. The simple equation is:
error = desired – guess
The desired is going to be either +1 or -1
The guess is going to be either +1 or -1
So the error will be either 0 (correct), -2 or +2
If the desired = +1, the guess = +1, the error = 0
If the desired = -1, the guess = -1, the error = 0
If the desired = -1, the guess = +1, the error = -2
If the desired = +1, the guess = -1, the error = +2
Learning Rate
The weights are adjusted each time to get the errors to zero as quickly as possible. The weights are only tweaked a little bit each time otherwise it wouldn’t learn anything. So it calculates the tweak by multiplying the error by the input. We we use the word tweak here but really you would call it delta weight because it is a small incremental change but tweak will do for now.
tweak = error times input
This tweak value is added to the weight for that particular input.
newWeight = weight + tweak
Which we can now cobble together as:
newWeight = weight + (error x input)
This can have profound effects on the tweaking of the weights and so we also multiply it by a learning rate (or constant). This will ensure that only small jumps or tweaks are made. So the equation will now look like this:
newWeight = weight + (error x input) x learning rate
Learning rates will often be values like 0.1 or 0.01, there is no hard and fast rule but if in doubt use 0.1. This is what is also called a hyperparameter.
Hyperparameters
The learning rate is something you can tweak, it will make a difference and so it is called a hyperparameter. In a neural networks we will look at other hyperparameters such as number of nodes, hidden layers etc but we are getting ahead of ourselves. However it is important to introduce key concepts and phrases at this stage so that they may make more sense later on.
Next article
We jump straight into coding, if you haven’t coded before or used p5.js then please go through the free eBook (PDF) to familiarise yourself or better still buy all the books in the series, ‘The Joy of Coding’, whatever!
Go to part 2
