Let’s code a Neural Network from scratch — Part 3
In this last section we’ll step away from simulating how the brain works to implement back propagation which bears no resemblance to the biological function of the brain.
We’re essentially going to use Newton’s method of optimisation to follow a gradient descent down towards the right answer. This is done by differentiating the answer you currently have:
We want to iterate through each neuron and adjust the weights so that the result you would achieve is closer to the desired result.
Let’s look at an example, in the case of card #1825 which is a 3 the ideal output layer would look like this:
All the weights are set to -1 except from the fourth along which is set to +1 indicating that the card is a 3. As expected, considering the weights were initially created randomly the actual answer is far off and our network is completely untrained. We got -0.9 as opposed to +1, therefore we want move along the sigmoid curve toward +1.
We can find the amount to move by looking are the error which is -1.9 (-1-0.9=-1.9). Which at this extreme is very low (1–0.9)(1+0.9) = 0.19. Lastly, we multiply it by a learning rate of 0.01 which just says we want to move slowly and not overshoot our goal. The total change is -1.9*0.19*0.01= -0.00361.
The learning is small at the beginning but we go through the weights according to this change: we move the weights according to the amount of input we receive from the layer before.
At the same time, though, I will also tell the layer below that it contributed to this error (a sort of apportionment blame passing back through the layers). This blame is in proportion to the weight currently in place.
Backpropagation
Now the neuron is capable of feeding back values to its inputs, I simply need to connect together the network so that it can go back-to-front through the layers, updating them:
Now the network is ready, all we need to do is to train the network. Then we can test it with random samples from the test set.
Training the network
Switching back to the main tab, we can create a piece of code that can train or test according to the mouse button that we select.
We’ve also added two buttons, one which trains the network using 500 randomly selected cards from the training set. The other which randomly selects one card from the testing set. Note that only the cards from the testing set contribute to the success rate.
There are many things we can visualise one of which are the connections between each neuron, there are too many to to draw them all but go through each neuron in the hidden and output layer and only draw the strongest connection. This doesn’t tell us very much but it’s interesting how their behaviour reflects the learning. They initially change rapidly to slowly come to an equilibrium and become stronger.
The success rate seems to be on average around “0.75” which isn’t bad considering its simplicity.
I hope you’ve found this article somewhat informative if so, be sure to follow for more computational design goodness. :)
If you’ve enjoyed this and want more please hit the 💚
Check out this report on ANN & visualising NMIST using a spring system on ISSUU
The original algorithm is from Alasdair Turner
