Behind the Eyeballs of AI: How Neural Networks Work

Sometimes it seems like practically every time I ask how a cool piece of new technology works, I get a two letter response: AI.

But then, if I try to ask how AI (artificial intelligence) works, I’m told that it’s complicated and takes a lot of linear algebra and multivariable calculus to understand.

For any curious person, this is annoying! It’s fun to look into the inner workings of complex systems that affect our lives. Being told you need to read two math textbooks to understand the basics of something is discouraging.

Fortunately, in the case of neural networks, it’s not true!

You definitely don’t need a deep understanding of linear algebra and multivariable calculus to understand the basics of neural networks. Now don’t get me wrong: complex math is essential for understanding neural networks at a deep level and creating your own neural networks from scratch.

But that’s not why most of us ask how AI works. If you just want to get a better feel for what’s going on behind the scenes of things people call “AI”, you don’t even need to know what linear algebra or multivariable calculus is.

Reading and understanding this article will make you more knowledgeable about neural networks than 99% of people. If you want to create your own neural networks and become more knowledgeable on the topic than 99.99% of people, then go learn some math.

After reading this article, you’ll be able to explain to anyone the structure of a neural network and how neural networks “learn” as they’re exposed to new data!

A neural network is the name for the computer program that’s the “brain” of an AI system. But before talking about neural networks, we’ll first look at what AI is, in a broad sense.

What even is AI?

It’s weird that it’s so hard to define a technology that’s supposed to be all around us.

Self-driving cars use AI to not run into stuff. Speech-to-text systems use AI to transcribe words. My phone’s handwriting keyboard uses AI to make sense of my sloppy pointer finger calligraphy.

While these might all seem like super different tasks, all these systems have a key thing in common.

Broadly speaking, an AI program is a computer program that finds patterns and does cool things with these patterns.

This doesn’t sound that impressive until you realize that humans pretty much just do the same thing.

Think about any complex, human-y thing that humans like to do. Let’s take having a conversation with someone. At a fundamental level, conversations work because each person finds patterns coming from the other person, then does cool things with these patterns. Each person’s ears and brain work together to find patterns of vibrations in the air. The brain then converts these vibration patterns into words with meanings, which is a pretty cool thing to do. Each person’s eyes and brain work together to find patterns of movements in the other person’s face. The brain then converts these tiny movements into emotional signals, which is also a pretty cool thing to do.

No matter how complicated a task is, any task can be reduced to finding patterns and using these patterns to do cool things. Of course, most tasks involve insanely large numbers of patterns, but they’re ultimately still based on patterns.

A self-driving car AI system learns to recognize objects on the road, like signs and road markings. The system then uses these patterns to construct an internal model of the road and steer safely.

Speech-to-text systems learn to recognize common speech sounds, and they combine patterns of these sounds into words. In other words, these systems are built to recognize patterns of patterns.

At its core, the challenge of making really good AI is trying to make systems that can accurately recognize patterns of patterns (of patterns of patterns of patterns…) in the real world.

If that idea makes sense to you, then you understand the essence of AI.

Types of AI

Now is also a good time to clear up the difference between two commonly discussed forms of AI: artificial narrow intelligence and artificial general intelligence.

Artificial narrow intelligence (ANI) is a pattern recognition based AI system that is designed to do a specific task. This is the kind of AI that’s all around us today. Examples of ANI include fingerprint scanners, automatic credit card fraud detection systems, and every other application of AI discussed above. These systems are built to recognize patterns in specific situations, and the state-of-the-art ones are damn good at it.

Artificial general intelligence (AGI) is where things start getting really crazy. AGI is designed to match or exceed the intelligence of a human. This means that an AGI system should be able to learn to recognize any pattern that a human could learn to recognize. As of 2020, AGI does not exist yet. Most experts agree that we’ll probably have AGI sometime between decades and centuries from now, but at the moment, we’re still working on it.

Here, we’ll be talking specifically about ANI. The general principles we’ll explore here would totally be relevant to an AGI system as well, but they’ve only ever been implemented in ANI.

Now that we understand the basics of AI, we can start getting into neural networks.

How neural networks work — broad overview

A neural network is the name for the computer program that’s the “brain” of an AI system. Neural networks are designed so that they get smarter as they see more and more data.

To understand this process of a neural network “getting smarter,” we’ll look at a classic application of neural networks: handwritten digit recognition.

The University of California, Irvine, assembled a famous dataset of handwritten numbers from 0–9. The dataset contains images of numbers volunteers wrote by hand, and these images are paired with the actual number each one is supposed to be.

To make it easier for normal people’s computers to work with, each image was scaled down to 8x8 pixel resolution. Here’s what the actual scaled-down images look like:

Not too hard for a person to figure out. Just stand about 20 feet back so the images get all blurry. It’s pretty clear that the first one is a 0, the second one is a 3, and the third one is a 7 written by some nerd who crosses their 7s.

But if I asked you to write a computer program to identity which is which, then that’d be a good challenge. That’s a perfect thing to use a neural network for.

Just like a human baby, a neural network can’t just recognize these images right away. First, we have to train the neural network.

As we talk about training, just pretend the neural network is a person. This makes the verbs in the rest of this section make a lot more sense. In the next section, we’ll get into how a neural network does all this neat stuff without actually being a person.

To train the neural network, we start by showing it an image and telling it what that image is supposed to be. So basically, “neural network, this thing is a 0.”

Then we do it again. “Neural network, this thing is a 3.”

And we keep doing that over and over again until we’ve shown it lots of images of every number. But don’t show it every image we have. The images and solutions that we show it are called the training data.

After it’s seen a bunch of images, we say that the neural network has been trained. The next step is to test the trained neural network.

To test the neural network, we show it one of the images that it hasn’t seen yet and ask the neural network what number it thinks it is. “Neural network, what do you think this is?”

If it identifies the image correctly, as a 5 in this case, we keep track of that. If it identifies the image incorrectly, we keep track of that too. We keep showing the neural network images that it hasn’t seen yet and making it identify them until we’ve run out of images. The images that we show the neural network during the test phase are called the testing data.

The neural network’s accuracy score is the percentage of images it correctly identifies. We want to get this neural network’s accuracy score as close to 100% as possible.

There are several parts of the neural network that we can adjust to improve the accuracy score, which we’ll explore later on. After we tweak the neural network a bit, we train it once again, then test it again, and calculate a new accuracy score. We keep repeating this process, changing the neural network a little each time, until we’re satisfied with how it performs!

For challenges like this one that involve classifying things, this is how basically all neural networks get smarter over time.

But there’s still one thing that you’re probably confused about, for good reason.

The neural network isn’t a person. How is it looking at stuff and identifying patterns like a person would?

Before we get into how the neural network identifies patterns, let’s first figure out how it “looks at” stuff at all.

How neural networks look at stuff

The pixelated numbers in this dataset are really just a bunch of light and dark colored boxes. We immediately see them as numbers, but that’s only because we have a really well trained neural network living inside our heads. For a neural network that doesn’t know anything yet, we need an easier way of representing these images.

Let’s say we have some random image like this:

If we wanted to, we could represent the shades of the boxes as numbers instead of colors. 0 could stand for white, 1 could stand for black, and numbers in between 0 and 1 could stand for shades between white and black. Here’s what that would look like:

[[0.5, 0.75, 0.5],
[0, 1, 0],
[0.5, 0.75, 0.5]]

If we know that this is supposed to be a 3x3 square image, we wouldn’t even need to put the numbers on separate lines. We could just make one list of numbers, like this:

[0.5, 0.75, 0.5, 0, 1, 0, 0.5, 0,75, 0.5]

This list is a lot easier for a neural network to handle! We could follow this same set of steps for any image, including the number images we already have. This means that the image of the number 5 above would look like this to a neural network:

[0.0, 0.125, 1.0, 1.0, 1.0, 0.875, 0.3125, 0.0, 0.0, 0.5625, 1.0, 0.6875, 0.375, 0.5, 0.1875, 0.0, 0.0, 0.5625, 1.0, 0.125, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1875, 1.0, 0.8125, 0.0625, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 1.0, 0.75, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.75, 1.0, 0.25, 0.0, 0.0, 0.0, 0.0, 0.0625, 0.6875, 1.0, 0.375, 0.0, 0.0, 0.0, 0.25, 1.0, 1.0, 0.5, 0.0, 0.0, 0.0]

Neural networks see images as a list of numbers, just like this.

Now that we understand how neural networks see images, we’ll look at the structure of a very simple neural network.

Simple neural network structure

A neural network is simply made up of neurons connected to other neurons. If every circle is a neuron, and every line is a connection, here’s what that looks like for a simple neural network:

This is called a feedforward neural network, and it’s the type of neural network we’ll explore here. There are tons of different types of neural networks, but feedforward neural networks are the simplest to understand.

Each of those neurons on the far left is called an input neuron. That neuron on the right is called the output neuron.

The word “neuron” sounds really fancy and sciencey, but a neuron is really just a box that holds a number.

This neural network might look pretty useless, but it’s actually very simple to input any one of our images into it.

First, increase the number of input neurons to the number of pixels in the image. Because it’s an 8x8 image, the number of pixels in the image equals 8 X 8, which equals 64. So we need 64 input neurons.

Remember that “list representation” of an image that we made above? Because each number in the list representation we made above represents a pixel, the list is also 64 numbers long.

From here, we can put each number in the list inside its own input neuron.

Each input neuron represents a single pixel of the image. And just like that, we’ve fed the neural network an entire image!

Of course, the neural networks needs to actually do something with the image to be useful.

Passing information between neurons

This is where the action starts. Don’t worry about the purpose of passing information between neurons just yet. We’ll talk about that in just a moment! For now, focus on understanding how information is passed between neurons.

To figure out what number should be in the output neuron, we add up all the input neurons connected to it.

As an example, let’s say a neural network with 5 input neurons looks like this:

To figure out what value should be in the output neuron, we add up all the input neurons that are connected to the output neuron. The result would look like this.

But addition alone isn’t quite enough for a working neural network. We also need to give each connection a certain weight.

When a connection between an input neuron and the output neuron is weighted, the input neuron’s number is multiplied by the connection’s weight before being added to the next layer. Let’s see that in action. Below, the numbers in green are the weights of each neuron’s connection.

Below, we can see the results of the multiplication in green as well.

The sum of the weighted input neurons is stored in the output layer.

So now we have two different parameters we in this system that we can mess with: the value of the input neurons, and the weights of each connection.

There’s just one more thing we need to do to this output neuron. All the input neurons have a value between 0 and 1, but the output neuron’s value right now is way above 1. To keep things consistent, we’ll put the output neuron’s number into a mathematical function called the sigmoid function that spits out a number between 0 and 1, no matter what the input is.

This way, the output neuron’s range always matches the input neurons’ range. This will become important later. Putting the output neuron’s value into the sigmoid function and replacing the output neuron’s original number with the result is called transforming the output neuron. In the context of neural networks, a function like this that transforms values is called an activation function.

This whole weighting process still probably seems pretty arbitrary. But weights are very useful, and you’re ready to understand why!

Super simple classification neural network

It’s actually possible to make a simple classification system with just the concepts we’ve already covered! Let’s say you want to make a system that can distinguish between the number 1 and the number 0.

To make seeing the boundaries of each image a little easier, I’ve added in border lines.

Looking at the 1 and the 0 next to each other makes a very important detail clear: 1s usually have very dark pixels in the middle of the image, while 0s usually have very light pixels in the middle of the image.

If we can get our neural network to recognize this, then we have a pretty good shot at classifying 1s and 0s correctly.

Fortunately, we have the perfect tool for this: weighting.

Each input neuron represents a pixel. The darker a pixel is, the closer its neuron’s value is to 1. The lighter a pixel is, the closer its neuron’s value is to 0. By weighting each input neuron’s connection to the output neuron, we can effectively give every pixel a weight. This is an important concept, so spend a second thinking about this to make sure it makes sense.

With weighting, we can make some pixels more important in calculating the output neuron’s value, and we can make some pixels less important.

Say we gave all the neurons that correspond to pixels in the middle of the image a positive weight. Let’s say we also gave all the input neurons that correspond to pixels around the edges of image a negative weight. This means that dark pixels in the middle of the image make the output neuron’s value go up, while dark pixels on the edges of the image make the output neuron’s value go down.

Now let’s imagine what would happen if we put an image of the number 1 into our weighted neural network. Images of the number 1 (like the one above) tend to have a ton of dark pixels in the middle of the image. Because the pixels are really dark, the numbers in the neurons that represent these pixels will be close to 1. We have these middle neurons’ connections weighted positively, which means that the value of the output neuron will be large and positive. Once the output neuron’s number has been transformed with the sigmoid activation function, the output neuron’s number will be super close to 1. (If the transformation aspect is confusing, take another look at the graph below.)

In contrast, let’s imagine what would happen if we put an image of the number 0 into our weighted neural network. Images of the number 0 tend to have more dark pixels at the edges of the image. Dark pixels on the edges of the image mean that the input neurons that represent the edge pixels will have values that are close to 1. Since these edge neurons’ connections are negatively weighted, the value of the output neuron will also be negative. Once the output neuron’s number has been transformed with the sigmoid activation function, the output neuron’s number will be close to 0.

With this system, images of the number 1 tend to make the output neuron’s value very close to 1, while images of the number 0 tend to make the output neuron’s value close to zero.

This gives us a simple classification rule.

If the output neuron’s value is quite close to 1, then the image is a 1. If the output neuron’s value is quite close to 0, then the image is a 0.

You may be wondering how the network defines “quite close” when it comes to the output neuron’s value. This is where training comes in! If we ran a bunch of pictures of 1s and 0s through this neural network, it would become clear that second layer neuron values that are larger than a certain number (not necessarily 0.5) mean that the image is probably a 1.

However, this super simple neural network structure has a major limitation. If we tried to test it with a new image, we’d get the number inside the output neuron, but the neural network wouldn’t actually make a guess. From here, another program or a person would have to look at the output neuron’s value and see whether it seems high enough to be a 1 or low enough to be a 0.

Fortunately, there’s a pretty simple solution to this issue.

Neural network with multiple output neurons

The neural network we just imagined weighted the middle pixel neurons’ connections positively and the outer pixel neurons’ connections negatively.

But there’s no reason we had to weight the middle neurons’ connections positively and the outer neurons negatively. If we wanted to, we could have weighted the outer neurons positively and the middle neurons negatively.

This inverse weighting would mean we get inverse results. An image of a 0 would cause the output neuron’s value to be close to 1, while an image of a 1 would cause the output neuron’s value to be close to 0.

If we wanted the neural network to automatically make a guess during testing without having to actually look at the output neuron’s value, we could put a neural network with our original weighting next to a neural network with inverse weighting.

We would put the same image into both neural networks. If the original-weighted neural network’s output neuron had a higher value, then the image would be classified as a 1. If the reverse-weighted neural network’s output neuron had a higher value, then the image would be classified as a 0.

Another way of expressing this idea is a neural network with two output neurons! This is simply putting the two neural networks above on top of each other.

If the top neuron’s value is higher, then the image is classified as a 1. If the bottom output neuron’s value is higher, then the image is classified as a 0.

It’s important to highlight that each neuron’s connection to each output neuron has its own weight.

It’s totally allowed for neuron A’s connection to neuron B to be weighted super heavily, even if neuron A’s connection to neuron C is weighted super lightly. Or A–B could be positively weighted, and A–C could be negatively weighted. Connections between two neurons are weighted, but neurons themselves are not.

This is closer to how real neural networks classify images. Each classification possibility has its own output neuron, and whichever output neuron has the highest number is the classification that the neural network guesses.

In this case, 1 and 0 each have their own output neuron. If the 1 output neuron’s value is higher, then the neural network guesses 1. If the 0 output neuron’s value is higher, then the neural network guesses 0.

If things are making sense so far, then you now understand the structure of a very simple binary classification neural network!

But there is an aspect of this neural network that we haven’t addressed in much detail.

How exactly do you set the weights of each connection?

We have the basic idea of how it’s done: depending on which output neuron we’re talking about, the connections between the middle input neurons and the output neuron we’re talking about are weighted either positively or negatively to identify the hole in the middle of the 0 or the line in the middle of the 1.

But really, that’s pretty vague. I mean, it’s not like somebody goes in and manually sets the weights for each connection. This would be tedious for even an 8x8 image, and it would be insanely hard for a 100x100 image!

The answer lies in a concept called the loss function.

The loss function

If our neural network were perfect, we know what we’d want it to do. If we showed it a 0, we’d want the 0 output neuron’s value to be exactly 1, and we’d want the 1 output neuron’s value to be exactly 0.

This would mean the neural network is telling us that it is sure it’s seeing a 0 when it actually is seeing a 0.

Of course, it’s basically impossible to reach this level of perfection. But it is possible to quantify how far away we are!

For each image, we do this by finding the difference between the ideal value and the observed value of each output neuron, then squaring this result. We add these squared values together to get our result.

The function that produces this result is called the loss function. People always want to minimize losses, and we want to minimize the result of our loss function to make the neural network as accurate as possible.

But how do we know how to minimize the loss function?

Let’s start by understanding the function that produces the observed value of one output neuron. Remember, this value is generated by multiplying each input neuron by the weight of its connection to the output neuron we’re working with, and adding these products all together. Using a bit of math, we could write that process like this:

And the ideal value is simple to find for each output neuron. If we want the test image to be classified as the output neuron’s image, the ideal value is 1. If we don’t want the test image to be classified as the output neuron’s image, the ideal value is 0. One reason we use the sigmoid function is so that we can make the ideal value a nice 0 or 1, instead of trying to take a wild guess.

If the ideal value is 1, we just subtract 1 from the equation above. The expanded form of the difference between the observed value and the ideal value in this case looks like this:

The loss function for a single output neuron is the square of this big thing. In this case, this is the loss function of the output neuron that indicates if the image represents a 1.

To get the loss function for an entire image, you repeat this process for each output neuron. If the output neuron you’re working with corresponds to the actual image shown, then the ideal value is 1. Otherwise, the ideal value is 0.

At the end, you add together the loss function for each output neuron.

This gives you the loss function for the entire image. The lower this value, the better the neural network did for the image.

But we’re not quite done. This is just the total loss function for one image.

This is where things start getting crazy.

To find the total loss for all images, we repeat this entire process for every image in the entire set of training data. At the end, we add together the loss functions for each image to get a single mind-bogglingly massive loss function for the whole neural network on the entire training set. The lower this gargantuan loss function is, the better the neural network is doing!

The reason we don’t just minimize the loss function for a single image is because we need to make sure the neural network can classify different kinds of images. If we just tried to minimize the loss function for a single image of the number 1, the neural network would find this easy. The best way for it to minimize this image’s loss function would be to simply classify every single image as a 1, no matter what. What a smartass.

We want to force the neural network to classify images as either 1s or 0s based on their appearance, so we add together the loss functions for each image to make the neural network take all images into consideration.

Now that we’ve explored what the loss function is, we can take a look at how the computer minimizes it.

Minimizing the loss function

The loss function that we just unpacked has three inputs:

1. The value of each input neuron
2. The ideal value of each output neuron (1 or 0, depending on the neuron)
3. The weights of each neuron’s connection with each output neuron

The value of each input neuron depends on the image the neural network is being fed, so we can’t change that. The ideal value of each output neuron also depends on the image, so we can’t change that either.

But we can do whatever we want to with the weights!

The weights are the component that the computer adjusts to minimize the loss function.

To understand a bit about how this works, let’s look at a 2D function.

Let’s say we start at a random point on this curve, and we want to find our way to the bottom.

But here’s the catch: we can’t actually see the rest of the function. We can see just enough to figure out whether the function is going up or down where we are, but no more.

That’s not a big problem, though. If we’re trying to get to the bottom, we can just hop down a little.

And if we’re not down yet, we just hop down again.

And again.

If we realized we hopped too far and found the curve pointing the other way like that, we know we hopped over the bottom, which is awesome! Then we take a smaller hop the other way.

At this point, we know we’re pretty close to the bottom.

The computer does something similar to minimize the loss function!

The computer adjusts one weight at a time, while leaving the others constant. On the graphs above, the vertical axis is the value of the loss function, and the horizontal axis is the value of the weight that’s being adjusted.

But just like the example above, the computer only calculates enough of the loss function to see the slope of the function.

That’s okay for the computer though! Just like we did, the computer uses the slope of this graph to decide which direction to adjust the weight. If the slope is really steep, the computer adjusts the weight a lot, and if the slope isn’t that steep, the computer adjusts the weight just a little.

After the computer adjusts one weight, it moves onto the next weight and repeats the same process. It keeps doing this for the weight of every single connection in the entire network!

After all the weights are adjusted, the computer looks at all of the training data again to figure out where the new weights put it on the loss function graph. The computer then minimizes adjusts the weights more to further minimize the loss function, creating another new set of weights.

This process continues, potentially thousands of times, until the programmers tell the computer to stop or the computer is satisfied that it’s minimized the loss function.

And this all happens automatically during a single training session!

In short, training a neural network involves adjusting the weights of each connection between input and output neurons to minimize the loss function.

If that sentence makes sense to you, then you have a solid grasp on the basics of neural network training!

There’s just one more piece of the puzzle we’ll add now.

Structure of a bigger neural network

So far, we’ve been working with a neural network that only has two output neurons: one that corresponds to an image of a 0, and another that corresponds to an image of a 1.

But by now, you may have guessed that it’s possible to add an output neuron for every possible number from 0–9.

This neural network operates just like the one we’ve been discussing! The only difference is that it can be trained on the entire dataset, and it can identify any number from 0–9.

When training begins, the neural network sets random values for the weights of each connection. Over multiple cycles of optimization, the neural network gradually adjusts the weights to minimize the loss function and maximize accuracy.

But there’s another, even cooler aspect of neural networks that we haven’t even discussed yet.

So far, we’ve been linking input neurons directly to neurons that try to identify an entire number.

But what would you guess happens if we put a whole line of neurons in the middle of the network? This line of neurons is called a hidden layer.

For each input neuron, the hidden layer acts like an output neuron. For each output neuron, the hidden layer acts like an input neuron.

A beautiful thing occurs here. Following the optimization process we discuss above, the neurons in the hidden layer naturally learn to identify subsections of numbers! These small subsections can be things like lines, curves, and dashes that work together to form a full number, just as pixels work together to form lines, curves, and dashes. Often, the subsections of numbers that hidden layers identify are shapes that look nothing like part of a number to the human eye, but rather have only subtle mathematical relevance to the shape of a complete number.

In other words, the neural network uses pixels to identify small parts of numbers. It then uses these small parts of numbers to identify entire numbers.

Let’s just stop and take a moment to realize how epic that is.

Unleashing the equations and optimization procedures we’ve discussed above naturally causes hidden layers of neurons to identify subsections of numbers. The laws of mathematics alone identify deep patterns, without any human intervention.

And you don’t have to stop at just one hidden layer. Two, three, or hundreds of hidden layers can be added to models for different results. Each hidden layer can also contain different numbers of neurons. The possibilities for messing around with hidden layers are endless.

If you’re wondering how a programmer can improve a model’s accuracy once it’s been trained, adjusting hidden layers is a common thing to try. (It’s also worth mentioning that there are dozens of completely different neural network structures, each with their own fascinating and unique characteristics.)

Hidden layers are the way in which neural networks learn to recognize patterns of patterns (of patterns of patterns of patterns…), as we discussed in the beginning of the article. In the digit classification problem, hidden layers let the neural network identify patterns of numbers from patterns of number subcomponents from patterns of pixels.

The possibilities for utilizing neural networks extend far beyond handwriting recognition. Neural networks can be relevant to any activity that involves recognizing patterns. They can be relevant to cooking, flying planes, building houses, optimizing supply chains, writing AI-generated papers, growing peas, teaching languages, and almost any other activity that you could possibly think of.

That’s why it seems like AI is in every single new technology you ask about.

Patterns are at the core of everything.

Note: If you want to learn more about neural networks and AI, there are tons of high-quality resources online. Andrew Ng’s coursera courses on machine learning and deep learning are great places to start.