Handwriting number recognizer with Flutter and Tensorflow (part V)

Published in

Flutter Community

11 min readDec 15, 2019

Welcome back and thanks for joining in this last post of the series! 🥳 You all should feel pretty proud of what we are building and how much we have done in the last article.

For those new to this series, you can find the first, second, third and fourth part of the series in the hyperlinks.

In the last article, you achieved to have a functional app that recognises with some degree of precision the digits you are writing on the screen. That’s awesome! But let’s admit it, it doesn’t really look great, does it?

In this last post of the series, we will add some information for the user, outputting what we believe is the value the user wrote and presenting a chart showing how confident we believe that value to be true.

At the end of the article, you should end up with a finished app that looks like this one:

Let’s get started!

The chart

For the chart, we first need to add the fl_chart dependency to our pubspec.yaml. This is a very nice charting library for Flutter, make sure you check some of their samples here. In any case, add this at the bottom of your dependencies:

fl_chart: ^0.5.1

We will be working on our RecognizerScreen class as we have all the UI for there, so let’s import the library we just added:

import 'package:fl_chart/fl_chart.dart';

And now we need to do 4 things:

A variable to hold the chart data
A function that builds the chart data based on our prediction
A place where to call that function
To show the chart in the UI

This is a very typical checklist of what you have to do when using a new library, everything usually falls onto the same: variable, method, method call, rendering.

The variable

For creating the variable, we are going to do it right underneath our AppBrain declaration:

List<BarChartGroupData> chartItems = List();

We are going to use a bar chart very similar to this one from the samples. We will layout it with more simple colours to keep it in line with our app but feel free to experiment with it and try to get some cool effects.

So what we need is a list of BarChartGroupData. I’m not going to get too specific into the chart details as I could easily fall into another blog post to cover all that 😅, but you can check the documentation in here if you are curious.

The function

Now we need a function that receives our predictions as an argument and transforms them into something that the chart can read. That something is going to be the variable we declared above chartItems.

So let’s create a private method that receives our predictions. I will place a default value of an empty list for the case when we call that function without arguments. The function will look as follows:

void _buildBarChartInfo({List recognitions= const []}) {
  // Reset the list
  chartItems = List();

  // Create as many barGroups as outputs our prediction has
  for( var i = 0 ; i<10 ; i++ ) {
    var barGroup = _makeGroupData(i, 0);
    chartItems.add(barGroup);
  }

  // For each one of our predictions, attach the probability 
  // to the right index
  for (var recognition in recognitions) {
    final idx = recognition["index"];
    if (0 <= idx && idx <= 9) {
      final confidence = recognition["confidence"];
      chartItems[idx] = _makeGroupData(idx, confidence);
    }
  }
}

We want to be able to show 10 bars in the chart, because we are trying to predict digits between 0 and 9, so the number we are using in that for loop is equal to the number of outputs we are trying to predict. But you are totally right, how are we hard coding that value in here? It does should be a constant! So create a constant called kModelNubmerOutputs and replace that 10 value with it.

As for the second for loop, if you remember, our prediction doesn’t give us an array with 10 elements in it. When the probabilities are zero or close to zero, the result from tensorflow just omits them as they are not relevant.

If you get to inspect the array we get back from the prediction, you will notice we can use the index value to assign it to our chart list. We are creating a new BarChartGroupData for each element we have a prediction on and replacing it at the right position of the list (so they are sorted).

You have probably noticed this weird if (0 <= idx && idx <= 9) in the code. For some reason, maybe a bug, sometimes the prediction would return an extraneous index that is out of bounds. I just placed that safe check to avoid it so you don’t see any unexpected crash.

Also, you have noticed we are calling a function that doesn’t exist yet _makeGroupData. I just divided the code in two methods for more clarity. The method _makeGroupDate will just create the BarChartGroupData element. Just copy and paste the following above the method we created right before:

BarChartGroupData _makeGroupData(int x, double y) {
  return BarChartGroupData(x: x, barRods: [
    BarChartRodData(
      y: y,
      color: kBarColor,
      width: kChartBarWidth,
      isRound: true,
      backDrawRodData: BackgroundBarChartRodData(
        show: true,
        y: 1,
        color: kBarBackgroundColor,
      ),
    ),
  ]);
}

I also moved some of the styling values to the constants file, so just copy and paste this over there and feel free to play with this values:

const Color kBarColor = Colors.blue;
const Color kBarBackgroundColor = Colors.transparent;
const double kChartBarWidth = 22;

Calling the function

Now stop scrolling down, I want you taking a break and check the code we have so far 👨🏻‍💻. After that, my challenge to you is to try to identify where should we call this new _buildBarChartInfo method from. I am calling it from three different places.

The answer is right after the codiging kitties gif 😸.

So there are three places where we should be calling this function:

Right after getting the results from the prediction:

onPanEnd: (details) async {
  points.add(null);
  List predictions = await brain.processCanvasPoints(points);
  setState(() {
    _buildBarChartInfo(recognitions: predictions);
  });
},

After cleaning our drawing:

floatingActionButton: FloatingActionButton(
  onPressed: () {
    _cleanDrawing();
    _buildBarChartInfo();
  },
  ...

When we initialize the state of the RecognizerScreen class:

@override
void initState() {
  super.initState();
  brain.loadModel();
  _buildBarChartInfo();
}

Do you remember that we were setting the argument that our function receives to have a default value? That’s so we can easily call it from any point on the lifecycle without the need of having an argument to place in it.

Rendering the chart

The last piece for having our charts is to actually show them to the user. Let me tell you this part is a bit of the ugliest ones as we need to copy a big chunk of declarative UI code.

Find the last Expanded element we have, the one that contains the footer, and replace it by the following:

Expanded(
  child: Padding(
    padding: EdgeInsets.fromLTRB(32, 32, 32, 16),
    child: BarChart(
      BarChartData(
        titlesData: FlTitlesData(
          show: true,
          bottomTitles: SideTitles(
              showTitles: true,
              textStyle: TextStyle(
                  color: Colors.black,
                  fontWeight: FontWeight.bold,
                  fontSize: 14),
              margin: 6,
              getTitles: (double value) {
                return value.toInt().toString();
              }),
          leftTitles: SideTitles(
            showTitles: false,
          ),
        ),
        borderData: FlBorderData(
          show: false,
        ),
        barGroups: chartItems,
        // read about it in the below section
      ),
    ),
  ),
),
Expanded(
  child: Container(),
  flex: 1,
)

We want to add that last Expanded with flex 1 at the bottom to give the view a bit of padding, so our clean canvas button doesn’t get in the way of the chart.

Now if you run the app, you should be seeing something like this:

It is still not the prettiest of the prettiest, but we will get there 😉.

Another challenge before moving on

You have probably observed that the complexity of our recnogizer screen has grown a bit by adding the chart code. It would be much better if the code related to the chart library would live somewhere else. My last challenge for you is to refactor this code into its own component (UI included). So that our screen doesn’t need to know the logic of how to create charts.

Telling the user what is going on

Now this should be the easy peasy part compared to the chart, I can tell you that we are really almost done! 🥳 Just bear with me a few lines more.

What we want to do here is to remove that red header space and set some labels to give information to the user, like: “Draw a number in here”, “I think it’s a 4!”, etc.

The strings

First, let’s set those strings in our constants file. We are going to omit localization for this post, but it is something I will talk about some other time.

const String kWaitingForInputHeaderString = 'Please draw a number in the box below';
const String kWaitingForInputFooterString = 'Let me guess...';
const String kGuessingInputString = 'The number you draw is';

The variables

Now let’s set two variables to hold our strings in the RecognizerScreen class, right underneath the chartItems one:

String headerText = 'Header placeholder';
String footerText = 'Footer placeholder';

The UI

Then we are going to show those labels in our UI. First replace the Text(‘Header’) widget we have for this:

Text(
  headerText,
  textAlign: TextAlign.center,
  style: Theme.of(context).textTheme.headline,
),

Then, a bit more complicated, for the bottom label, do you remember the Expanded widget that contains our chart? Well, we need that Expanded inside a Column, inside a Container, inside another Expander. In case you don’t know this yet, use alt/option + enter to wrap a widget within another widget. It needs to look like this:

Expanded(
  child: Container(
    child: Column(
      crossAxisAlignment: CrossAxisAlignment.stretch,
      mainAxisAlignment: MainAxisAlignment.center,
      mainAxisSize: MainAxisSize.max,
      children: <Widget>[
        Expanded(
          child: Padding(
            padding: EdgeInsets.fromLTRB(32, 32, 32, 16),
            ...

I also added some column alignments that you will have to paste yourself.

Finally, we just add the footerText variable as the first children of the column (in its own Text widget) and we center it with another widget:

children: <Widget>[
  Center(
    child: Text(
      footerText,
      style: Theme.of(context).textTheme.headline,
    ),
  ),
  Expanded(...

If you run the app, you will see it doesn’t look at its best yet…

The text from the footer is way too close to the drawing area and there’s a big gap between the chart and the bottom of the screen. Let’s fix it adding a padding to the Expanded widget that wraps the Column widget with the chart:

Expanded(
  child: Padding(
    padding: const EdgeInsets.fromLTRB(0, 32, 0, 64),
    child: Container(
      child: Column(...

Also we can remove that last Expanded widget with flex 1 that we have at the bottom. It won’t be useful longer as we have our padding. Run the app again, it should look much better this time.

One last touch, remove the red color background we have given to the header, we don’t need the color highlighting any longer.

Assigning values to the text variables

The last thing we need is to assign variables to those. First we will create a method that sets the default value for this labels:

void _resetLabels() {
  headerText = kWaitingForInputHeaderString;
  footerText = kWaitingForInputFooterString;
}

And then we will call this method twice, once in initState and another when pressing the clean button:

void _cleanDrawing() {
  setState(() {
    points = List();
    _resetLabels();
  });
}...@override
void initState() {
  super.initState();
  brain.loadModel();
  _buildBarChartInfo();
  _resetLabels();
}

We are only missing to show a different text when we make the prediction, so let’s create another method for that right under _resetLabels:

void _setLabelsForGuess(String guess) {
  headerText = ""; // Empty string
  footerText = kGuessingInputString + guess;
}

And we need to call this method right when we have the information about the prediction, so add it to the onPanEnd callback this way:

onPanEnd: (details) async {
  points.add(null);
  List predictions = await brain.processCanvasPoints(points);
  setState(() {
    _setLabelsForGuess(predictions.first['label']);
    _buildBarChartInfo(recognitions: predictions);
  });
},

If you run the app at this moment, the labels should be working perfectly, give it a try!

One last thing…

Do you notice anything weird with our UI during that last run? The header area is way too big for the content it needs to show. On the other hand, the footer area is a bit too small and the chart barely has vertical space.

If you inspect the code, you will see tha the first Expanded widget that we have in our scene has a flex value of 1, and that we don’t have a flex value set in the Expanded widget that is our footer. To solve our spacing problems, we only need to give a flex value of 3 to this last Expanded widget, so that the structure looks as follows:

Run the app one last time and see that everything now fits in its place.

You made it again!

This is a great achievement! 👏🏻 Our app is completely done, take a moment to reflect on it. It took us only 5 articles to build a machine learning model, a mobile application that can run on any platform (iOS and Android) and use our model in the app to predict what the user is writing with his fingers.

Now you have the tools to build mobile apps with machine learning models; or to create ML models to run in mobile applications, or both! There are so many things you could experiment on with this app skeleton from now on. For instance, you could try to recognize alphanumeric values rather than just numbers; you could try to improve the accuracy of the model and see how it works; you could try to run the prediction (inference) every X points drawn so it looks as it is been done in real time, etc. But in all this sentences there’s a you could try. It is really cool what you can learn following tutorials such as this one; but it is impressive how much you discover when you try on your own and get to play with this technologies. Let me know in the comments if you build something new from this, I am really curious about it 🙂

As usual, you can access all the code from this section here.

sergiofraile/when_flutter_meets_tensorflow_part_5

A new Flutter application. This project is a starting point for a Flutter application. A few resources to get you…

github.com

This has been the end of this series but not my last article about mobile and machine learning 😉, so I’m really looking forward to see you in the next one!