Build Your First Neural Network From Scratch

Let’s teach AI to do something simple

Pavel Ilin
Nov 25, 2020 · 5 min read
stylized diagram of neural networks in the brain
stylized diagram of neural networks in the brain
Image by Ahmed Gad from Pixabay

I have been wanting to play with neural networks for a very long time and finally I found a window of opportunity to mess around with neural nets. It’s pretty far away from Skynet, and I don’t think that I fully grasped the math behind this, but let’s teach AI to do something simple first.

Theory Behind the Code

Neural networks are not a new concept. They were first introduced by Warren McCulloch and Walter Pitts in 1943.

We are going to build a single-layer neural net without hidden layers or a perceptron. It will consist of an input layer with training examples, synapses or weights, and neurons, and an output layer with correct answers. This is a graphical representation of the neural net:

Image for post
Image for post

Also, we need an understanding of some math concepts such as sigmoid and derivatives to know how the neurons are learning. But what the neurons are doing is simply taking an input value, multiplying by synapse weight. Then we sum all these multiplications and use the sigmoid function to get output in the range between 0 and 1.

Neuron representation:

Image for post
Image for post

Sigmoid function:

Image for post
Image for post

Problem Definition

We have sequences of numbers on the input layer. And we want the neural network to return 1 if the first number of the input data set sample is 1 and return 0 if the first number is 0. This is represented in the output layer. And this is how the problem set looks:

table showing 5 examples of how the first input determines the final output
table showing 5 examples of how the first input determines the final output

Prerequisites

I hope we develop some level of conceptual understanding and we can start coding.

We are going to use Python and NumPY library.

NumPY installation:

pip install numpy

If it’s installed successfully, we can proceed to the coding part. First what we need to do is to import NumPy to the Python file:

import numpy as np

Now we are ready to describe and train our neural network.

Training

First, let’s create a sigmoid function:

Image for post
Image for post

Next, we will define training examples, inputs (which are in a four-by-five matrix), and output:

Image for post
Image for post

Next, we need to initialize synaptic weights by generating random values and put them in the shape of a four-by-one matrix:

Image for post
Image for post

Next, let’s start building a training model. We will use a for loop, and all training will happen inside this loop. We will invoke the sigmoid function and pass the sum of all inputs multiplied by sigmoid weights. Np.dot will do matrix multiplication for us. Here is what we have:

Image for post
Image for post

And what we have as output:

Image for post
Image for post

Now we are ready to do the training. We are going to do it by calculating the difference between the output we have and the actual output of the sigmoid function. Then we can adjust our weight accordingly to the severity of the error. We are going to repeat this several times, 10,000 times, for example.

Let’s define the sigmoid derivative:

Image for post
Image for post

And this is how we calculate and adjust weights:

Image for post
Image for post

Let’s start learning and see how the results will be affected by the length of the learning process. We will start with 100 iterations:

Image for post
Image for post

Not bad for the start — our AI learned how to recognize patterns, but the percentage of errors is still pretty high. Now let’s do 1,000 iterations:

Image for post
Image for post

Much better, but let’s keep going and do 10,000 iterations:

Image for post
Image for post

And finally 100,000 iterations:

Image for post
Image for post

We can keep up with adding more learning iterations, but we will never get to 100% accuracy because that would require an infinite amount of computation. But even for the worst case, we have a probable accuracy of 99.77%, which is pretty good.

For the final code, I’ve made it look nice and separated it by functions. I also added a very sophisticated method of storing weights in a text file. This allows us to do the learning process once, and when we need to use our AI, we can just import weights and use the sigmoid function.

Conclusion

Our first artificial intelligence is ready for production. Even though it can only recognize very simple patterns on a very small data set, now we can expand it and, for example, try to teach the AI to recognize something in pictures. I will write about further development in the next article.

Keep learning, keep growing!

Better Programming

Advice for programmers.

Sign up for The Best of Better Programming

By Better Programming

A weekly newsletter sent every Friday with the best articles we published that week. Code tutorials, advice, career opportunities, and more! Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Thanks to Zack Shapiro

Pavel Ilin

Written by

Software Engineer, AI researcher and Transhumanist.

Better Programming

Advice for programmers.

Pavel Ilin

Written by

Software Engineer, AI researcher and Transhumanist.

Better Programming

Advice for programmers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store