Neural Networks implementation from the ground up part 2 — Feedforward

Welcome back to the series of Neural Networks implementation from the ground up. In this part, we’ll be going through how to implement the feedforward flow of the neural network.

In the last blog, we went through how to setup the neural network structure along with the basic building block — Matrix

The what?

“Feedforward” basically calculates the output of the neural network based on it’s weights and biases. Note that no learning happens in this stage. Learning is part of the backpropagation flow.

In the last blog, we saw this example of a neural network.

How will it’s feedforward look like with the given weights and biases?

For layer 0’s first neuron:

(1)

Similarly, for layer 0’s second neuron:

(2)

And finally for last neuron:

(2)

You get the point, it’s the weighted sum along with biases.

If you look closely, we can neatly represent all of the outputs in matrix multiplication form:

As this is for the first layer, we do not apply any activation function for i as it is the input. Let’s see how the second layer’s output will look like:

for 1st neuron:

here, g(x) denotes the activation function, and in this example, I’ll be using the sigmoid function.

This helps us to map a value between 0 and 1, no matter how big/small the input x is. Prevents the previous layer’s actual outputs to affect this layers outputs from going too much out of scale.

Hence for the 2nd layer, our equation looks like this:

And finally for layer 3:

Code

As we have already written our utilities for Matrix in the previous section, these operations will be straightforward.

We start by creating a utility which will help in initialising the NeuralNetwork object:

public class NNBuilder {
    public static NeuralNetwork create(
            int inputRows, int outputRows, List<Integer> hiddenLayersNeuronsCount) {
        List<Matrix> weights = new ArrayList<>();
        List<Matrix> biases = new ArrayList<>();

        int nHiddenLayers = hiddenLayersNeuronsCount.size();
        for (Integer integer : hiddenLayersNeuronsCount) {
            biases.add(Matrix.random(integer, 1, -1, 1));
        }

        // last layer's biases
        biases.add(Matrix.random(outputRows, 1, -1, 1));

        int previousLayerNeuronsCount = inputRows;
        for (int i = 0; i < nHiddenLayers; i++) {
            weights.add(
                    Matrix.random(
                            hiddenLayersNeuronsCount.get(i), previousLayerNeuronsCount, -1, 1));
            previousLayerNeuronsCount = hiddenLayersNeuronsCount.get(i);
        }
        weights.add(Matrix.random(outputRows, previousLayerNeuronsCount, -1, 1));

        return NeuralNetwork.builder()
                .weights(weights)
                .biases(biases)
                .layers(hiddenLayersNeuronsCount.size() + 1)
                .build();
    }
}

And for our specific neural network, we can call the builder in this way:

NeuralNetwork neuralNetwork = NNBuilder.create(5, 2, List.of(3, 3)));

Coming to feedforward, that too is pretty straightforward:

public void feedforward(Matrix input) {
      List<Matrix> layerOutputs = new ArrayList<>();

      // first input is without any activation function
      Matrix bias = biases.getFirst();
      Matrix weight = weights.getFirst();
      Matrix outputLayer1 = bias.add(weight.cross(input));
      layerOutputs.add(outputLayer1);
      Matrix prevLayerOutput = outputLayer1;

      for (int i = 1; i < getLayers(); i++) {
          input = prevLayerOutput.apply(Functions::sigmoid);
          bias = biases.get(i);
          weight = weights.get(i);
          Matrix outputLayerI = bias.add(weight.cross(input));
          layerOutputs.add(outputLayerI);

          prevLayerOutput = outputLayerI;
      }
      setLayerOutputs(layerOutputs);
  }

Where Functions::sigmoid is defined separately:

public class Functions {
    
    private Functions(){}

    public static double sigmoid(double x) {
        return 1 / (1 + Math.exp(-x));
    }

    public static double differentialSigmoid(double x) {
        return sigmoid(x) * (1 - sigmoid(x));
    }
}

differentialSigmoid will be used when we implement backpropagation.

For doing one iteration of feedforward, we simply call

neuralNetwork.feedforward(input)

Each layer output is stored in layerOutputs which will later be used in the backpropagation stage.

Example

Lets assume we have input of 5 binary bits, and we need to classify whether these are divisible by 3 or not. Output 0 will indicate how much likely it is divisible by 3, and output 1 will indicate how much it is not. We will be walking through a proper MNIST example once we cover all concepts.

    public static void main(String[] args) throws IOException {
        List<Pair<Matrix, Matrix>> trainingData = List.of(
                Pair.of(new Matrix(new double[][]{{0, 1, 1, 1, 0}}).transpose(), new Matrix(new double[][]{{0, 1}}).transpose()), //14, not divisible
                Pair.of(new Matrix(new double[][]{{0, 1, 0, 0, 1}}).transpose(), new Matrix(new double[][]{{1, 0}}).transpose()) //9, divisible
        );

        NeuralNetwork neuralNetwork = NNBuilder.create(5, 2, List.of(3, 3));
        for(Pair<Matrix, Matrix> p : trainingData){
            neuralNetwork.feedforward(p.getA());
            Matrix outputLayer = neuralNetwork.getLayerOutputs().getLast();

            System.out.println("expected: \n"+p.getB()+"\noutput: \n"+outputLayer);
        }
    }

Pair is a utility class for pairing up data structures. Here it is

The output is pretty random, which is expected as we have set all weights and biases to random values:

expected: 
[0.0]
[1.0]

output: 
[-0.08266908882116297]
[1.5195448278508819]


expected: 
[1.0]
[0.0]

output: 
[-0.10386840368442773]
[1.496952583483225]

In the next blog, we will be seeing how to nudge these random weights and biases such that the output moves closer to what we expect, i.e how will the network actually learn.

Resources

1. https://medium.com/@satviknema/neural-networks-implementation-from-the-ground-up-part-1-f1a392016010