A gentle introduction to a neural network. Starting with the essential, thought experiment.

A Neuron or Perceptron is a simplified mathematical model of a biological neuron. The perceptron algorithm was invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt. Initially and strangely it was intended to be a machine, rather than a program. It was reckoned to be used for image recognition.

We have myriads of libraries in our toolkit, some popular ones like TensorFlow — SciKit, that we practically would never want to worry about how the fundamental unit of a neural network works.

Nevertheless, for me, it has always been imperative to learn how the inner functioning of a system works. Or simply, how does any system work. Whether or not I can build the system is another story. Knowing how several elements of a system interact with each other, how a simple data as input comes out as a fine prediction without any hard-coded logic. Self-driven. How does a neural network do that and what does a single neuron contribute to the network? These were some questions that kept me curious for a long time.

It turns out that a neuron isn’t as complex as we imagine it to be. And it isn’t that straightforward and simple either. 😁

So now is the time we grab a cup of coffee. ☕️

We will start with a simple game and down the line, slightly far ahead, we will wrestle our way out of calculus(next part of the series). But believe me, it wouldn’t be so daunting as it sounds. And I really hope by the end of the post, you will feel more inquisitive and enthusiastic about machine learning rather than believing it to be a mind-bending piece of a subject. Let’s start…

Enter Game — Hi-Lo

The aim of the game is to improve our guess and end up with an accurate score. Simple enough! Also, I know the answer and at every guess, I will let you know if your guess is low or high.

Let’s say I ask you how many kilograms make a pound? I am sure you know the answer already.

For a moment, let’s forget that we know the magical figure and act like we are 5. So what’s your guess? 1.8? Okay.. we have to start somewhere and 1.8 is good enough start.

Okay, before proceeding, we should also remind ourselves that the relationship between kilogram and pounds is linear. That means if we double the kilogram or half it, the same effect is proportional in pounds. And, can we agree upon a relationship, which can be expressed as…

Now, at this moment we know what M is! That’s right, it’s your guess! Let plug the value of M= 1.8 and let’s find out how many pounds is 80 kg.

Great! Now, I pull my card out and it says Low. Time to bump the guess.

Awesome! Now, I pull my card out again and it’s Low. Time to refine further…

Fantastic! But this time the guess is High.

Bravo! 2.2 is a pretty reasonable guess. And this process can be replicated for the decimal part for accuracy. But for now, let us focus on the essence of prediction and correction.

If you followed closely, we updated our guess(M) at every step. We were making a judgment to either, increase/decrease a certain value of/from M, based on whether our guess was low or high.

Is there a way we can represent this mathematically? And more importantly, what can we do with the error(the difference in the actual output and the prediction) in prediction to improve our guess(M). Let us see how can we correlate error and guess?

The Equation of a Line

Let’s visualize our relationship for lbs & kg in a generic way. It can be expressed as follows:

Can you make a guess what is the significance of this equation? It is an equation of a line. You can see below the graph of this equation when m is 2.2. The line coincides at 0,0 which is good and at 10lbs the value for kg is 4.

However, the complete equation of line involves an additional term which is denoted by letter b, the y-intercept of the line. The updated equation becomes:

Our earlier equation was simpler because when kilogram is 0 even the pounds is 0.

An example where we can consume “b” is with the relationship between Celsius-Fahrenheit because, at 0° C, Fahrenheit is 32°. So even when the product of m * x is 0, y-term has a value of 32.

The following graph can give a good picture of what the b term signifies. You can see that X and Y do not coincide at 0, which is because of the addition of b. You also notice that at roughly 32° F, Celsius is 0, which is about right and at 41° F, Celsius is 5. So I hope b-term is slightly clear now.

We will soon utilize the equation of a line in the next part of this series. For now, let’s take a step back.

Mathematizing Hi-Lo

Our guesses were logical & smart. But how could we formulate a process? Before we move further let’s just come on the same page with the following terminology:

• Target is the actual answer/value
• Prediction is what the guess was
• The Error is the difference between Target & Prediction.

We also need a name for the value that we either added or removed at every step. Delta is also used commonly to denote change. Let’s call it Δm. So now our equation for finding our target value becomes:

In every step, we either increased or decreased the value of m. Which implies that these 2 entities must be combined together. Hence, we add Δm and m. When Δm is negative the total value of the equation would reduce and when Δm is positive it would increase. So our initial equation started with just m, with the next step it changed to m + Δm. Moving on…

In our case…

What is Target?

› For the right value of ∆m, (m + ∆m) * x, should give us the correct value.

What is Guess?

› We knew this from our initial steps and it’s m * x

What is Error?

› We know it already and after substituting the above individual terms, we get…

And that my friend, is the correlation between the error and our change. Let’s rewrite this:

So now we have come with a result that will help us write a program which will arrive at accurate answers. Let’s test this out…

This is insane, isn’t it? 💥

In a single step, we arrived at an accurate answer. This works well because we had an accurate answer to our conversion problem. Many a time, accurate answers are not available. For example, we cannot know the answer for a classifier problem beforehand, we could only guess whether the dividing line for our data-set is good or bad. This will get crystal clear when we explore the next part of this series.

We will build upon the relationship, ∆m= E / x, further to solve intriguing problems like a classifier.

Lastly, in our human approach to the guess, we couldn’t arrive at the prediction in a single step. Similarly, we would want our program to approach the right answer gradually. And the reason we want to do that is to avoid overshooting the target. Overshooting the target becomes extremely evident when we apply the relationship for a classifier. We want our network to make small adjustments to improve the prediction holistically. Or in other words, we want our program to converge to a reasonable value. And essentially this is what most of the machine learning algorithms do, predict — improvise — predict.

This brings us to the end of the first part of the series. In the next part, we will learn about learning rate, summation function & activation function. We will use JavaScript & a bit of React(for canvas rendering only) that will eventually turn out to be a classifier.

If you enjoyed this article, please 👏 this article to share it!

Follow me (Meher Ranjan) on Medium to stay connected for the next part of this series. 👍

Cheers 🍻