Handwriting number recognizer with Flutter and Tensorflow (part I)

Sergio Fraile
Oct 6, 2019 · 8 min read
Image for post
Image for post

There is a nice intersection between machine learning and front end development, but we often find overwhelming to gain knowledge in both areas and get them to play together. My intention is to solve that with a couple of short articles where we will dive into the basics of creating an ML model and how can we apply it to a mobile app built in Flutter.

What are we going to build?

Image for post
Image for post

This will involve several steps that I’ll break down in different articles to keep them short. Some of the things we are going to do are:

  • Create a tensorflow model to recognise handwritten numbers.
  • Create a flutter application with that model and a finger paint canvas.
  • Process the drawing into an image that we can feed into our model.

Also we will mention some of the common pitfalls when feeding real world data into a model.

For this first article, we are going to focus only in creating a model in Tensorflow and exported to Tensorflow Lite so we can use it in our mobile app.

Remember that you will be able to see the full code for this section at the end of each article.

Let’s get started

But before getting hands on, you must know there are two ways you can follow up this first article. You can either run it on your local machine with Python or you can use a web Jupiter notebook for doing so. My advice is that if you are not familiar with python and pip, you should use the Jupiter notebook to avoid messing with your local machine configuration.

To run it locally

If you don’t want to read the Tensorflow documentation, fear you must not, this is what you really need in order to install it.

# For python 2
pip install tensorflow
# For python 3
pip3 install tensorflow

To run it on a Jupiter notebook

If you choose to use a notebook, you won’t need to install tensorflow, so you can skip everything until importing the model.

The project structure

your_project_directory/
> models/
> numbers_mnist.py

At the moment models will be an empty folder where we’ll save our models later. Make sure to create that folder.

Writing our machine learning model

Import Tensorflow

import tensorflow as tf

Load the dataset

mnist = keras.datasets.mnist

If you don’t know about the MINST dataset, you can find more information here. But basically it is a set of images that represent hand written numbers. A couple of important take aways from this data set are:

  • Images consist of 28x28 pixels.
  • The digit is represented in a 20x20 centred image with a 4 pixel padding (that is how we get the 28x28 mentioned before).
  • The images are greyscale, where the background color is black and the drawing line is white.
Image for post
Image for post

Then we can load the data assigning it directly to our training and testing variables:

(x_train, y_train),(x_test, y_test) = mnist.load_data()

x_train and x_test contains the MNIST images, and the respective y variables contains the labels of those images, or in other words, if that image represents a 3, or a 4, etc.

Normalizing

x_train, x_test = x_train / 255.0, x_test / 255.0

Normalizing is the process of changing this values to a common scale. In this case we are transforming the 256 color values (from 0 to 255) into values between 0 and 1.

This is useful as it means we can accept any normalized data. As an example, imagine we are building this model and accepts images where each pixel have a color value between 0 and 255. A couple fo weeks later we want to be able to use the same model with a 16 color value, but we found the input size doesn’t match. Normalizing this values between 0 and 1 would make this two inputs compatible.

Model

model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])

Doesn’t matter too much how complex or simple is a model, from a front-end point of view; but what is really important for front end devs is the input shape from the first layer/line and the output from the last layer/line. This marks what type of input we are going to need to feed to any model we work on and what the model is going to return to us. We can easily infer that from any model without diving into the complications of the layers inside.

Compile

model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

When compiling the model we need to define the optimizer and the loss function. We are not going to dive in detail on those two values here, there are better specialised websites where you can learn about them. We also define the accuracy of the model as a metric.

Fit

When training our model, we are going to tell Tensorflow to run the training a certain number of times, known as epochs. For instance, we could tell Tensorflow to train our model for 100 epochs.

There’s the possibility that we may reach our desired accuracy (or another metric) before reaching the last epoch, and so we can use something call *callbacks* to stop the training process.

This is particularly useful as training a model can take several hours or days, but don’t worry, the model we are building here will train quite fast. As an estimate, each epoch takes 2 seconds running on my Mac’s CPU. Also, we will be aiming for a 99% accuracy, and we should reach that around the epoch 20.

We will define our callback as follows:

class myCallback(tf.keras.callbacks.Callback):
def on_epoch_end(self, epoch, logs={}):
if(logs.get('acc')>0.99):
print("\nReached 99.0% accuracy so cancelling training!")
self.model.stop_training = True

For the callback above, if you are using Tensorflow 2, make sure to use ‘accuracy’ rather than ‘acc’.

Once we have our callback, we can fit our model this way:

model.fit(x_train,
y_train,
epochs=50,
callbacks=[myCallback()])

As you have notice, the callbacks argument of the fit method is an array, so we could have different callbacks running at the same time.

When running this code, you’ll be able to see how the loss and the accuracy are changing for each epoch, feel free to spend as much time as you want digging into this logs:

Epoch 15/25
60000/60000 [==============================] - 2s 32us/sample - loss: 0.0322 - acc: 0.9890
Epoch 16/25
60000/60000 [==============================] - 2s 32us/sample - loss: 0.0304 - acc: 0.9897

Evaluate

Evaluating the model is as easy as calling this function:

model.evaluate(x_test, y_test)

During the evaluation phase, we’ll also be able to see how our model is performing:

10000/10000 [==============================] - 0s 25us/sample - loss: 0.0782 - acc: 0.9800

A few takeaways to reflect on in here is that the size of our testing data is way smaller than our training data, therefore this evaluation also runs pretty fast in our case. Second, you’ll notice that the accuracy and the loss don’t really match with what we achieved during training, this is totally normal as we are facing data never seen by our model before.

We don’t really need to evaluate the model for the porpoise of this tutorial, but is always good to see how your model is performing. You can always come back here, create a different model and see how it performs against this one.

TFLite

Save

# tensorflow 1.x
model.save('models/my_mnist_model.h5')

If you are using Tensorflow 2, you could save the Tensorflow model (rather than the keras model) this way:

# tensorflow 2.x
tf.saved_model.save(model, "./models/mnist")

Convert

# tensorflow 1.x
converter = tf.lite.TFLiteConverter.from_keras_model_file(
'models/my_mnist_model.h5'
)

In case you used the saved_model method from Tensorflow 2, you could load the model into the converter this way:

# tensorflow 2.x
converter = tf.lite.TFLiteConverter.from_saved_model('models/mnist')

Then we convert the model:

tflite_model = converter.convert()

Save again

open("models/converted_mnist_model.tflite", "wb").write(tflite_model)

You made it!

As mentioned before, this is how all this code together should look like:

You can access all the code from this section in here. Looking forward to see you in the next section!

Flutter Community

Articles and Stories from the Flutter Community

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store