LeNet-5 in 9 lines of code using Keras

Keras is a high-level open source APIs, written in Python and capable of running on top of TensorFlow, Microsoft’s CNTK, or Theano

Even though TensorFlow introduced in v1.0 some high level APIs, Keras is still a really good option and a concise way to quickly write and experiment with Machine Learning models.

We will use here LeNet-5 network to train a model on MNIST dataset that identifies handwritten digits.

It is always a good to do some data exploration before we start using it, find outliers, and decide if we need a preprocessing phase to uniform or augment it. And also to make sure that all the classes are covered by or more or less the same number of samples. We do not need to follow this step here given that we know that MNIST is good to use as is.

Now let us discuss the model structure, we will use LeNet-5 which comprises of 7 layers, not counting the input layer as you can see in the graph below.

LeNet-5 Architecture. Credit: LeCun et al., 1998

MNIST images are 28x28 pixels which is smaller than what LeNet-5 expects 32x32 pixels. An easy solution to that is just to pad the images with zeros to bring the MNIST images size up to 32x32 pixels.

LeNet-5 layers:

1. Convolution #1. Input = 32x32x1. Output = 28x28x6 conv2d
2. SubSampling #1. Input = 28x28x6. Output = 14x14x6. SubSampling is simply Average Pooling so we use avg_pool
3. Convolution #2. Input = 14x14x6. Output = 10x10x16 conv2d
4. SubSampling #2. Input = 10x10x16. Output = 5x5x16 avg_pool
5. Fully Connected #1. Input = 5x5x16. Output = 120
6. Fully Connected #2. Input = 120. Output = 84
7. Output 10

A Keras implementation of LeNet-5 network would be extremely readable and maintainable. We can change it and experiment with it with ease.

model = keras.Sequential()

model.add(layers.Conv2D(filters=6, kernel_size=(3, 3), activation='relu', input_shape=(32,32,1)))
model.add(layers.AveragePooling2D())

model.add(layers.Conv2D(filters=16, kernel_size=(3, 3), activation='relu'))
model.add(layers.AveragePooling2D())

model.add(layers.Flatten())

model.add(layers.Dense(units=120, activation='relu'))

model.add(layers.Dense(units=84, activation='relu'))

model.add(layers.Dense(units=10, activation = 'softmax'))

Checkout this notebook which contains all the code required to build and train LeNet-5 using Keras APIs and also visualize the training.

As a bonus I want to show you how much it would take to implement the same model using TensorFlow low-level APIs.

Unlike Keras, the first thing you need to calculate is the input and output shapes for all the different model layers.

n_classes = 10

# Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
mu = 0
sigma = 0.1    

weights = {
    # The shape of the filter weight is (height, width, input_depth, output_depth)
    'conv1': tf.Variable(tf.truncated_normal(shape=(5, 5, 1, 6), mean = mu, stddev = sigma)),
    'conv2': tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma)),
    'fl1': tf.Variable(tf.truncated_normal(shape=(5 * 5 * 16, 120), mean = mu, stddev = sigma)),
    'fl2': tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma)),
    'out': tf.Variable(tf.truncated_normal(shape=(84, n_classes), mean = mu, stddev = sigma))
}

biases = {
    # The shape of the filter bias is (output_depth,)
    'conv1': tf.Variable(tf.zeros(6)),
    'conv2': tf.Variable(tf.zeros(16)),
    'fl1': tf.Variable(tf.zeros(120)),
    'fl2': tf.Variable(tf.zeros(84)),
    'out': tf.Variable(tf.zeros(n_classes))
}

# Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.
conv1 = tf.nn.conv2d(x, weights['conv1'], strides=[1, 1, 1, 1], padding='VALID')
conv1 = tf.nn.bias_add(conv1, biases['conv1'])
# Activation.
conv1 = tf.nn.relu(conv1)
# Pooling. Input = 28x28x6. Output = 14x14x6.
conv1 = tf.nn.avg_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

# Layer 2: Convolutional. Output = 10x10x16.
conv2 = tf.nn.conv2d(conv1, weights['conv2'], strides=[1, 1, 1, 1], padding='VALID')
conv2 = tf.nn.bias_add(conv2, biases['conv2'])
# Activation.
conv2 = tf.nn.relu(conv2)
# Pooling. Input = 10x10x16. Output = 5x5x16.
conv2 = tf.nn.avg_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

# Flatten. Input = 5x5x16. Output = 400.
fl0 = flatten(conv2)

# Layer 3: Fully Connected. Input = 400. Output = 120.
fl1 = tf.add(tf.matmul(fl0, weights['fl1']), biases['fl1'])
# Activation.
fl1 = tf.nn.relu(fl1)

# Layer 4: Fully Connected. Input = 120. Output = 84.
fl2 = tf.add(tf.matmul(fl1, weights['fl2']), biases['fl2'])
# Activation.
fl2 = tf.nn.relu(fl2)

# Layer 5: Fully Connected. Input = 84. Output = 10.
logits = tf.add(tf.matmul(fl2, weights['out']), biases['out'])

Checkout this notebook which contains all the code required to build and train LeNet-5 using TensorFlow low-level APIs.

References:

1. https://colab.research.google.com/drive/1CVm50PGE4vhtB5I_a_yc4h5F-itKOVL9
2. https://colab.research.google.com/drive/1kV3Jpxzup63GfJB1FGKxTSKd6Ek8J3sA

Would you be interested in learning more about this https://leanpub.com/ml-mobile