Understanding Vectorized Implementation of Neural Networks
Studying the inner workings of neural networks can be difficult for beginners without clear visualization of data movement inside the network.
I’ve created a simple neural network example architecture and detailed matrix operations to aid in understanding the inputs, outputs and inner details of neural networks. The diagrams below and the matrix operations help to study the vectorized implementation of feed-forward propagation algorithm(used for predictions) and backpropagation algorithm(used to train the neural network).
Notation :
A Simple Network Architecture :
The network consists of 3 layers: an input layer, a hidden layer, and an output layer.
It has 5 output classes and accepts an input vector (x) (feature vector) with 4 features.
Feed Forward Propagation (Using the network for predictions) :
The feed-forward propagation takes an input feature vector (x) and outputs predicted classes (hΘ)corresponding to each training example, here we have 3 training examples.
Note: In our example, the bias unit (b), which is stated in other literature, is substituted by appending 1 as the first element in the feature vector and every activation layer (a) , and adding a corresponding θ₀ in the weight matrix.
The backpropagation (training the neural network) :
The backpropagation algorithm calculates the error matrix (Δ) and the gradient matrix (D) corresponding to every weight matrix (Θ). The gradient matrix can be used by optimization algorithms to optimize the weights of the network.
