INTRO TO BACKPROPAGATION

Yusufaolodo
Nov 4 · 4 min read
Neural Network

Neural Networks provide a lot of service to data science, machine learning in particular. Although there are various algorithms used in training a model.I am going to explain the rationale behind Backpropagation. To understand Backpropagation, it is a paramount to know what a Multi-layer Neural Network (MNNs) is which is a type of Feed Forward neural network.

Artificial neural network can mimic the learning ability of its biological counterpart and is used as an approach to Machine Learning. Machine learning involves adaptive mechanisms that enable computers to learn from experience and/or by example. The most popular approaches to machine learning include artificial neural networks and genetic algorithms.

Two key similarities between biological and artificial neural networks

  1. The Learning process involves connections between neurons to determine the function of the network. So learning of both networks means the modification of these inter-neuron connections.
  2. Network Structure :building blocks of both networks are simple computational devices(although artificial neurons are much simpler than biological neurons)but these simple devices are highly interconnected

A multi-layer neural network is a feed-forward neural network with one or more hidden layers. The network consists of an input layer of source neurons, at least one middle or hidden layer of computational neurons, and an output layer of computational neurons.

MULTI-LAYER NEURAL NETWORK

A hidden layer “hides” its desired output. Neurons in the hidden layer cannot be observed through the input/output behaviour of the network. There is no obvious way to know what the desired output of the hidden layer should be.

Learning in a multi-layer network proceeds the same way as for a perceptron (single neuron network). The network computes its output pattern, and if there is an error −or in other words a difference between actual and desired output patterns −the weights are adjusted to reduce this error. A training set of input patterns and desired output patterns is presented to the network(Supervised Learning). The limitations of a perceptron is its inability to learn and identify non-linear patterns due to the use of step and sign function as its activation function as there is no gradual weight modification to the input signals.

A Perceptron

Multi-Layer neural network employs sigmoid activation function which is a non-linear function enabling the neural network to grasps complexities such as circles and curves.

BACK-PROPAGATION LEARNING

In a back-propagation neural network, the learning algorithm has two phases:

  1. First, a training input pattern is presented to the network input layer. The network propagates the input pattern from layer to layer until the output pattern is generated by the output layer.
  2. If this pattern is different from the desired output, an error is calculated and then propagated backwards through the network from the output layer to the input layer. The weights are modified as the error is propagated.
SIGMOID FUNCTION

BACK-PROPAGATION TRAINING ALGORITHM

Example of a Multi-Layer Neural network with Two Neurons

Step 1: Initialisation -Set all the weights and threshold levels of the network to random numbers uniformly distributed inside a small range.The weight initialisation is done on a neuron-by-neuron basis.

Step 2: Activation -Activate the neural network by applying inputs x1(p), x2(p),…, xR(p) and calculate the actual outputs at the end of each layer.

Step 3: Weight training
Update the weights in the network layer by layer backwards by propagating backwards the errors associated with output neurons.

Weight Training and updating the weights in the Neural Network for Output Layer
Weight Training and updating the weights in the Neural Network for Hidden Layers

Step 4: Iteration -Increase iteration p by one, go back to Step 2 and repeat the process with the next training example p+1 until the selected error criterion is satisfied.

In a multi-layer neural network, the first hidden layer will be able to learn some very simple patterns. Each additional hidden layer will somehow be able to learn progressively more complicated patterns. Scientific American infographic down below details how complex a neural network gets the further the layer goes.

source- Scientific American
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade