How neural networks actually work

You might have come here after reading an article on AI by Practicum by Yandex. Good choice! Let’s dig deeper into neural networks.

What is a neural network

A neural network is a kind of database. It stores data (basically, numbers) and can move this data between its cells. The only difference is the structure. In a regular database, cells are connected in rows and columns. In neural networks, cells are connected… you know… like in a network:

But the point of a neural network isn’t in the data that it stores. Most of the time, the cells are empty anyway. The point of a network is in the connections between the cells. See those arrows between the cells? That’s the most critical thing in a network.

As data moves through the network, it gets transformed. The arrows define how every bit of data transforms. In other words, you feed this network some numbers, the network does math to these numbers according to the arrows, and you get some numbers on output. That’s it. The neural network is just a lot of fancy math.

Example

In our last article, we gave an example of a delivery robot that crosses the street. Say, our delivery robot has to identify a car. It has a camera. A camera gives us an image that might have a car in it, or it might not. An image is mostly a pile of pixels, tiny dots with information about color. For simplicity, let’s say that our robot only sees in shades of grey, so for every pixel it sees, there is a number between 0 and 255. 0 is black, 255 is pure white, and everything in-between is shades of grey.

Again, for simplicity, let’s say our robot has a low-resolution camera that captures images that are 32 x 32 pixels, which is 1024 pixels per image. It’s a pretty small image, but for now, it will do. To us, these pixels make an image, and despite the low resolution, we can somehow identify a car (even guess the model). But to a computer, it’s just a bunch of numbers that represent shades of grey.

How does a computer tell which image is of a car? Can we create an algorithm that checks sequences of numbers against some template? Can we make a template that says: ‘These numbers definitely represent a car’? We can’t. Cars come in all shapes and sizes, and our template can’t account for all possible cars.

What we can do is this:

1. Get a bunch of photographs of different cars, of which we definitely know: yes, these are cars.
2. Maybe even a bunch of photographs of stuff that’s not cars. Mark them as ‘not cars.’
3. Take the data from these photographs and put them into a database. For 1000 images we’ll have a little over 1 million numbers (1000 x 1024 = 1,024,000)
4. Do some fancy math to that million numbers (we won’t go there just yet). Remember that computers are good at doing math to millions of numbers.
5. As a result of that fancy math, we’ll have a very, very, VERY complicated formula.
6. This formula takes in 1024 numbers and outputs a number between 0% and 100%.
7. Magically, to us, this number between 0% and 100% represents how likely any given image is an image of a car.

So, what does this formula have to do with the network chart? This chart represents the fancy formula. You input the numbers into the input nodes, and the output gives you another number. This number says, ‘I am this much confident this data stands for a car.’

This chart is just an oversimplified representation of the logic behind the fancy math that goes on in step 4. In reality, you’ll have 1024 nodes on the left, and two or three ‘hidden layers’ in-between. There will be millions, possibly billions of connections between nodes, and trillions of mathematical operations.

All to identify a car in a small black-and-white photograph.

Why the network?

Network is just a graphical representation of the math that happens under the hood. In reality, it’s just math: billions or sums, subtractions, multiplications, and divisions. The only problem is that to us; it would make no sense, so we drew a network-style chart to try and understand what was going on.

What is going on is this: we have 1024 numbers that need to be summed and multiplied about a billion times to give us one final number. This is all there is.

What does it hide in the hidden layers?

As for the hidden layers — it has to do with the way this fancy math is done. We’ll cover it in-depth a future article, but for now, suffice to say that:

1. Humans do not program the fancy math of a neural network — a machine finds it through trial and error.
2. This process of trial and error is called machine learning: this is how a neural network kind of learns proper math to tell a car from a cat.
3. The machine needs to store its knowledge about this math somewhere. For that, it uses the hidden layer. It’s just a bunch of storage space for data and formulas.
4. The problem with the hidden layer is that it is generated and regenerated randomly in the process of learning, and to an external observer, it will be just random noise.
5. Humans can look into the hidden layer and even tinker with it, no problem. But it was designed for the machine to store temporary learnings, so there are no user-serviceable parts there.

In this aspect, it’s much like the human brain: we can cut it open and see what’s inside, but it will make no sense to us, with the level of complexity it has.

So, is the neural network a brain?

It is based on similar principles, but it’s not a brain. Here is what it has in common with the human brain:

It’s performance-oriented. The aim of a neural network is to give some useful results (for example, tell a car from a cat). The way it achieves that is of less importance. Its internal logic can be all messed up, as long as it gives an accurate enough result.

A neural network is like a muscle: it can do stuff, it can be trained, but it’s not specifically designed for a task. It just trains to perform that task and gets better at it with time.

It can learn (or train). There is an iterative process that shapes the connections in a neural network to make it produce useful results. Unlike a brain, however, it is not curious. A programmer is the one who makes it learn.

It’s unclear how it works. We understand the basic math behind a neural network, but once it starts machine-learning, we’re no longer so sure what it does. And we don’t need to, as long as it performs.

And here is how a neural network is different from a brain:

It’s just a file. All the formulas and the data that a neural network processes can all be saved as a file and copied onto a thumb drive — or emailed. It’s just some data and formulas.

It has a different topology. A neural network can have an unlimited number of connections. A physical neuron in a brain can only have so many. This affects how neural networks calculate and what results they create.

Compared to a human brain, it’s dead simple. There’s that. Current neural networks, even the most advanced and high-powered, don’t even come close to the size and complexity of a human brain.

Ok, how does a neural network learn?

Glad you asked. Join us next time for an in-depth on that.