Handwriting number recognizer with Flutter and Tensorflow (part I)

Published in

Flutter Community

8 min readOct 6, 2019

There is a nice intersection between machine learning and front end development, but we often find overwhelming to gain knowledge in both areas and get them to play together. My intention is to solve that with a couple of short articles where we will dive into the basics of creating an ML model and how can we apply it to a mobile app built in Flutter.

What are we going to build?

We are going to build a handwriting number recognizer for flutter. So if you follow this series, you should end up with an app looking like this:

This will involve several steps that I’ll break down in different articles to keep them short. Some of the things we are going to do are:

Create a tensorflow model to recognise handwritten numbers.
Create a flutter application with that model and a finger paint canvas.
Process the drawing into an image that we can feed into our model.

Also we will mention some of the common pitfalls when feeding real world data into a model.

For this first article, we are going to focus only in creating a model in Tensorflow and exported to Tensorflow Lite so we can use it in our mobile app.

Remember that you will be able to see the full code for this section at the end of each article.

Let’s get started

As mentioned before, we are going to focus on creating a small model with Tensorflow and export it. It’s alright if you don’t have enough experience with Python or Machine Learning, that’s why we are dividing this into small blocks so you have an overall understanding of whats going.

But before getting hands on, you must know there are two ways you can follow up this first article. You can either run it on your local machine with Python or you can use a web Jupiter notebook for doing so. My advice is that if you are not familiar with python and pip, you should use the Jupiter notebook to avoid messing with your local machine configuration.

To run it locally

For running it locally, you need to have python running in your machine and installing Tensorflow. If you don’t know how to install Tensorflow, you can follow the instructions here. But before you install it, considering using a virtual environment to have an isolated working environment, so we don’t have to worry if things break. I have an article about virtualenv here, check it out!

If you don’t want to read the Tensorflow documentation, fear you must not, this is what you really need in order to install it.

# For python 2
pip install tensorflow# For python 3
pip3 install tensorflow

To run it on a Jupiter notebook

You can use a notebook, basically running python on a server through your browser. This is a very handy tool for people that works in this field as it allows you to reload certain parts of the code without running all of it again. Some of the more popular options are Colab by Google and Azure Notebooks by Microsoft. This options are free and will allow you to speed up the training of your model running it using a GPU.

If you choose to use a notebook, you won’t need to install tensorflow, so you can skip everything until importing the model.

The project structure

Once you are all set to go, let’s create a project folder for our small machine learning model, in my case I’ll use this structure:

your_project_directory/
    > models/
    > numbers_mnist.py

At the moment models will be an empty folder where we’ll save our models later. Make sure to create that folder.

Writing our machine learning model

Import Tensorflow

You can open numbers_mnist.py with your favourite editor and start by importing Tensorflow. If you are using a notebook, this is from where you need to start coding.

import tensorflow as tf

Load the dataset

Tensorflow already has some datasets prepared so you don’t have to download them and process them, which makes the task at hand way easier. We can load our MNIST data set as follows:

mnist = keras.datasets.mnist

If you don’t know about the MINST dataset, you can find more information here. But basically it is a set of images that represent hand written numbers. A couple of important take aways from this data set are:

Images consist of 28x28 pixels.
The digit is represented in a 20x20 centred image with a 4 pixel padding (that is how we get the 28x28 mentioned before).
The images are greyscale, where the background color is black and the drawing line is white.

Then we can load the data assigning it directly to our training and testing variables:

(x_train, y_train),(x_test, y_test) = mnist.load_data()

x_train and x_test contains the MNIST images, and the respective y variables contains the labels of those images, or in other words, if that image represents a 3, or a 4, etc.

Normalizing

We need to normalize the image values:

x_train, x_test = x_train / 255.0, x_test / 255.0

Normalizing is the process of changing this values to a common scale. In this case we are transforming the 256 color values (from 0 to 255) into values between 0 and 1.

This is useful as it means we can accept any normalized data. As an example, imagine we are building this model and accepts images where each pixel have a color value between 0 and 255. A couple fo weeks later we want to be able to use the same model with a 16 color value, but we found the input size doesn’t match. Normalizing this values between 0 and 1 would make this two inputs compatible.

Model

And create a basic sequential model, we don’t want anything too complex or that would take too long to train for the porpoise of this tutorial. In this case we are using an input layer which matches the size of the mnist dataset images, followed by a dense layer, a dropout layer with a 20% dropout rate and finally the last dense layer. This last layer has only 10 units that corresponds with the amount of outputs we are looking for our model (numbers from 0 to 9).

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

Doesn’t matter too much how complex or simple is a model, from a front-end point of view; but what is really important for front end devs is the input shape from the first layer/line and the output from the last layer/line. This marks what type of input we are going to need to feed to any model we work on and what the model is going to return to us. We can easily infer that from any model without diving into the complications of the layers inside.

Compile

We need to compile the model we just defined with the following method:

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

When compiling the model we need to define the optimizer and the loss function. We are not going to dive in detail on those two values here, there are better specialised websites where you can learn about them. We also define the accuracy of the model as a metric.

Fit

The concept of fitting our model means to train our model with the training data from the mnist dataset we loaded at the beginning.

When training our model, we are going to tell Tensorflow to run the training a certain number of times, known as epochs. For instance, we could tell Tensorflow to train our model for 100 epochs.

There’s the possibility that we may reach our desired accuracy (or another metric) before reaching the last epoch, and so we can use something call *callbacks* to stop the training process.

This is particularly useful as training a model can take several hours or days, but don’t worry, the model we are building here will train quite fast. As an estimate, each epoch takes 2 seconds running on my Mac’s CPU. Also, we will be aiming for a 99% accuracy, and we should reach that around the epoch 20.

We will define our callback as follows:

class myCallback(tf.keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs={}):
    if(logs.get('acc')>0.99):
      print("\nReached 99.0% accuracy so cancelling training!")
      self.model.stop_training = True

For the callback above, if you are using Tensorflow 2, make sure to use ‘accuracy’ rather than ‘acc’.

Once we have our callback, we can fit our model this way:

model.fit(x_train,
          y_train,
          epochs=50,
          callbacks=[myCallback()])

As you have notice, the callbacks argument of the fit method is an array, so we could have different callbacks running at the same time.

When running this code, you’ll be able to see how the loss and the accuracy are changing for each epoch, feel free to spend as much time as you want digging into this logs:

Epoch 15/25
60000/60000 [==============================] - 2s 32us/sample - loss: 0.0322 - acc: 0.9890
Epoch 16/25
60000/60000 [==============================] - 2s 32us/sample - loss: 0.0304 - acc: 0.9897

Evaluate

After we have a trained model, we want to evaluate it against the testing data. If you remember, when we loaded the MNIST dataset, we loaded both training and test data. This test data is use to see how good our model performs against data that it hasn’t been used during the training phase and therefore is totally new for our model.

Evaluating the model is as easy as calling this function:

model.evaluate(x_test, y_test)

During the evaluation phase, we’ll also be able to see how our model is performing:

10000/10000 [==============================] - 0s 25us/sample - loss: 0.0782 - acc: 0.9800

A few takeaways to reflect on in here is that the size of our testing data is way smaller than our training data, therefore this evaluation also runs pretty fast in our case. Second, you’ll notice that the accuracy and the loss don’t really match with what we achieved during training, this is totally normal as we are facing data never seen by our model before.

We don’t really need to evaluate the model for the porpoise of this tutorial, but is always good to see how your model is performing. You can always come back here, create a different model and see how it performs against this one.

TFLite

At this point we have a model compiled and what we want is to create a version for Tensorflow lite so we can use it on our mobile. This process has two steps: save our model and then convert it.

Save

We’ll save the keras model in our models folder with the following line:

# tensorflow 1.x
model.save('models/my_mnist_model.h5')

If you are using Tensorflow 2, you could save the Tensorflow model (rather than the keras model) this way:

# tensorflow 2.x
tf.saved_model.save(model, "./models/mnist")

Convert

Once the model is saved, we can convert it with a few lines of code. First, we create a TFLite converter and load the previously saved model with it:

# tensorflow 1.x
converter = tf.lite.TFLiteConverter.from_keras_model_file(
'models/my_mnist_model.h5'
)

In case you used the saved_model method from Tensorflow 2, you could load the model into the converter this way:

# tensorflow 2.x
converter = tf.lite.TFLiteConverter.from_saved_model('models/mnist')

Then we convert the model:

tflite_model = converter.convert()

Save again

And finally save our new TFLite model into the models folder:

open("models/converted_mnist_model.tflite", "wb").write(tflite_model)

You made it!

Congratulations! That last saved model is the one we’ll port into our Flutter application in the next tutorial.

As mentioned before, this is how all this code together should look like:

You can access all the code from this section in here. Looking forward to see you in the next section!