Intuition behind perceptron: the building blocks of Neural Networks

1. Introduction

Perceptron in deep learning is one of the most fundamental concepts that every data scientist is expected to master. A perceptron is a type of artificial neuron used in machine learning, particularly in binary classifiers. Perceptrons are fundamental to understanding more complex neural networks, as they introduce the concepts of weighted inputs and training via adjustment of those weights.

2. Origin

The perceptron’s algorithm was invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt, funded by the United States Office of Naval Research. It was intended to be a machine, rather than a program, and while its first implementation was in software for the IBM 704, it was subsequently implemented in custom-built hardware as the “Mark 1 perceptron“. This machine was designed for image recognition: it had an array of 400 photocells, randomly connected to the “neurons“. Weights were encoded in potentiometers, and weight updates during learning were performed by electric motors.

3. Components of a Perceptron

Fig: Diagrammatic representation of Perceptron(Image generated by Author)

1. Inputs (x₁, x₂, …, xₙ): These are the features or signals sent into the perceptron. Each input corresponds to a feature of the data being processed. For example, in a task to identify whether an image contains a cat, inputs might include various pixel values or derived features from the image.
2. Weights (w₁, w₂, …, wₙ): Each input feature has an associated weight that adjusts during the learning process. Weights are crucial as they determine how much influence each input has on the final decision. The weights are typically initialized randomly and are tuned during training to minimize prediction errors.
3. Bias (b): The bias is an additional parameter in the perceptron used to adjust the output along with the weighted sum of the inputs. The bias allows the activation function to be shifted to the left or the right, which may be critical for successful learning.
4. Summation Function (Σ): This function computes the weighted sum of the inputs, adding each input multiplied by its corresponding weight. The bias is also added to this sum. Mathematically, it can be expressed as ∑𝑖=1𝑛𝑤𝑖𝑥𝑖+𝑏∑i=1n​wi​xi​+b, where 𝑛n is the number of inputs, 𝑤𝑖wi​ are the weights, 𝑥𝑖xi​ are the inputs, and 𝑏b is the bias.
5. Activation Function: After computing the weighted sum, the perceptron passes this sum through an activation function, which makes the decision about the output. The simplest form of activation function used in a basic perceptron model is the Heaviside step function, which outputs a 1 if the input sum is above a certain threshold (usually 0) and 0 otherwise. For more details on Activation function, recommend reading this great article.
6. Output (y): This is the final binary result produced by the perceptron after the activation function has been applied. It represents the class or category as determined by the algorithm, based on the input data.

Each component plays a crucial role in how a perceptron processes input and learns from data. The interplay of these elements allows the perceptron to perform binary classification tasks, forming the foundation of more complex neural network architectures.

4. Similarity with Neurons

From the above, we can say that the model takes in some input, processes it, and generates an output. This is similar to what happens in a biological neuron.

Biological neurons (source — wallpaperflare)

The perceptron is conceptually similar to a biological neuron in several fundamental ways, which is why it is often referred to as an artificial neuron. Here’s how a perceptron mirrors the structure and function of a biological neuron:

1. Input Reception: Just as biological neurons receive signals from other neurons through dendrites, perceptrons receive multiple input signals. In the perceptron, these inputs are typically numerical values representing features of the data.
2. Weights and Synaptic Strength: In a biological neuron, the strength of the signal being transmitted from one neuron to another depends on the strength of the synaptic connections. Similarly, in a perceptron, each input has an associated weight that determines how much influence that input will have on the output. These weights are akin to the synaptic strengths in biological neurons.
3. Summation: A biological neuron sums the electrical signals received through its synapses, a process mimicked by the summation function in a perceptron, which computes a weighted sum of the inputs.
4. Activation: Biological neurons use an activation function — the process by which the neuron decides whether to fire based on the summed input it receives. If the input exceeds a certain threshold, the neuron fires (sends an output signal along its axon). Similarly, perceptrons use an activation function (such as a step function) to determine their output based on the weighted sum of inputs. If the total input exceeds a defined threshold, the perceptron outputs a specific value (commonly 1 or 0).
5. Output Transmission: When a biological neuron fires, it sends a signal through its axon to other neurons. In a perceptron, the output is sent as the result of its computations, influencing subsequent computational stages or serving as the final output of a model.
6. Adaptability through Learning: Both biological neurons and perceptrons have the ability to learn, though the mechanisms differ. In biological neurons, learning occurs through changes in the synaptic strengths, a process influenced by neuroplasticity. Perceptrons learn by adjusting the weights and sometimes the bias during the training process, guided by algorithms that minimize the error in output.

These similarities make the perceptron model a useful abstraction for understanding how networks of neurons might work together to process complex information in the brain, albeit in a highly simplified form.