Handwriting number recognizer with Flutter and Tensorflow (part III)

As usual, amazing to have you back!

For those new to this series, you can find the first and second part of the series in the hyperlinks.

In this third article, we are going to build the finger painting widget for our application. Hope you are ready for painting some serious stuff at the end of the tutorial 😉

Drawing on the screen

Let’s start checking what Flutter has available for us for painting in the screen. If we dig a little bit, we will find references to CustomPaint, CustomPainter and Canvas. If you are interested, bellow are the links to the Flutter documentation of this classes, but right above it I’m highlighting as best as possible the main differences.

CustomPaint class

CustomPainter class

Canvas class

I’m going to keep the long story short in here and resume it in a few bullet points:

• CustomPaint is a widget that provides us with a Canvas under the hood.
• CustomPaint uses a CustomPainter to run paint commands on the Canvas.
• We cannot easily access the Canvas value of a CustomPainter outside its methods.
• CustomPainter can be subclassed and we could override some of its methods , but it is difficult to replace the Canvas there.
• A Canvas cannot be used as a Widget by itself.
• A Canvas has a method to export an image of its content.

Real time images from one of our readers

I know, it may be a bit too much to process. Let’s see what we actually need:

We want a something that allow us to paint in a rectangle in the screen and allow us to export it as an image, so we can feed that image to our model.

Well… it does seem that neither Canvas nor CustomPaint can help us by themselves… 😭 I guess we will have to use both of them!

For the moment, let’s just worry to have a widget where we can use our finger to paint, so let’s bring CustomPaint onto our app. This is going to bring a little bit of code into our app, so let’s go step by step.

CustomPaint

First of all, replace our SizedBox by the following code:

CustomPaint(
  size: Size(kCanvasSize, kCanvasSize),
  painter: DrawingPainter(
    offsetPoints: points,
  ),
),

CustomPaint is already included in material.dart, so we don’t need to import anything else. You will notice that there are a couple of undefined things there: kCanvasSize, DrawingPainter and points. Let’s get them next.

Points

Points is simply a property declaration in our class and it is going to host a list of points. Just place it under our _RecognizerScreen definition as follows:

class _RecognizerScreen extends State<RecognizerScreen> {
  List<Offset> points = List();
  
  ...
}

kCanvasSize

Anything that starts by k is going to be a constant within our app, and that means we need to create a file to hold our contants, so create a constants.dart file and add the canvas size to it:

const double kCanvasSize = 200.0;

Then you will need to import our constants file to our recognizer screen as follows:

import 'package:handwritten_number_recognizer/constants.dart';

DrawingPainter

The drawing painter is going to be our custom subclass for CustomPainter. CustomPainter is used by CustomPaint to execute drawing commands in the canvas.

Our intention will be to pass a list of points to our CustomPainter and those will be the drawing commands we want it to execute. I will explain later how we are going to obtain that list of points.

So let’s create a new file called drawing_painter.dart and copy this code in there:

import 'package:flutter/material.dart';
import 'package:handwritten_number_recognizer/constants.dart';
final Paint drawingPaint = Paint()
  ..strokeCap = StrokeCap.square
  ..isAntiAlias = kIsAntiAlias
  ..color = kBlackBrushColor
  ..strokeWidth = kStrokeWidth;
class DrawingPainter extends CustomPainter {
  DrawingPainter({this.offsetPoints});
  List<Offset> offsetPoints;
  @override
  void paint(Canvas canvas, Size size) {
    for (int i = 0; i < offsetPoints.length - 1; i++) {
      if (offsetPoints[i] != null && offsetPoints[i + 1] != null) {
        canvas.drawLine(offsetPoints[i], offsetPoints[i + 1], drawingPaint);
      }
    }
  }
  @override
  bool shouldRepaint(DrawingPainter oldDelegate) => true;
}

You will also notice there is a Paint class at the top of the page, that is, as its name indicates, the paint we are going to use for painting in the canvas; and its values are defined in our constants file. Just add the following there:

const double kStrokeWidth = 12.0;
const Color kBlackBrushColor = Colors.black;
const bool kIsAntiAlias = true;

You have to import flutter/material.dart to been able to use Colors within that file.

Ok, let’s take a break and try running the app.

If you have run the app, you probably have noticed it doesn’t do anything yet. We have our green square and when we try to finger paint on it, nothing occurs. That is because we are not passing any drawing commands to our DrawingPainter.

What we need to do know is to capture our finger on the screen and pass that down to our DrawingPainter. We are going to use the points list we defined before for passing the data, but how are going to capture the data?

GestureDetector

To capture the finger movement when dragging it around the canvas area, we are going to use a GestureDetector.

Of course because nothing we do could be easier, we are going to wrap this GestureDetector widget onto a Builder widget! This is so much fun 😁

Let’s replace our CustomPaint widget by this monster:

Builder(
  builder: (BuildContext context) {
    return GestureDetector(
      onPanUpdate: (details) {
        setState(() {
          RenderBox renderBox = context.findRenderObject();
          points.add(
              renderBox.globalToLocal(details.globalPosition));
        });
      },
      onPanStart: (details) {
        setState(() {
          RenderBox renderBox = context.findRenderObject();
          points.add(
              renderBox.globalToLocal(details.globalPosition));
        });
      },
      onPanEnd: (details) {
        setState(() {
          points.add(null);
        });
      },
      child: CustomPaint(
        size: Size(kCanvasSize, kCanvasSize),
        painter: DrawingPainter(
          offsetPoints: points,
        ),
      ),
    );
  },
),

Don’t feel overwhelm and let’s explain what we are doing here. You will see the onPan methods above, what we are basically doing is recording from the moment the user puts the finger in our CustomPaint until he/she releases. We mark the release moment setting the point as null.

Now the little problem we have is that the GestureDetector is giving us the coordinates based on the whole screen of the device; but we want them localized for our CustomPaint area, so our coordinate system should be values between 0 and kCanvasSize (that we defined as 200).

For transforming this global coordinate into local we are required to find the GestureDetector render area and apply a transformation (that the function globalToLocal does for us).

Now, why do we need to use a Builder widget here? Short answer (if there’s such a thing for this topic) is that, normally, when building widgets, the BuildContext type parameter is propagated to our child widgets automatically. Basically we propagate the app context all the way down. However, we only have a single declaration of context usually at top of our widget class, for instance:

@override
Widget build(BuildContext context) {
  return Scaffold(

But how do we capture the context at the scope of a particular widget? Well, we pass it through the Builder widget, and now we have a new context at the scope of that Builder we just declared. Therefore finding the RenderBox from our Builder widget won’t return the same one that at the top level.

There is another way of doing this using keys, as Diego Velasquez describes in this great article Flutter: Widget Size and Position. Feel free to experiment with both approaches and let me know your thoughts in the comments.

If we run the app again now, we should be able to finger paint on our canvas. You also may notice that we can get out of the painting box 😂 Let’s fix that real quick.

Last touches

In order to wrap it up for this article, let’s do two more things: avoiding to paint outside the box and decorate the painting area.

Clipping

There are several ways we can tackle down the issue of having points outside our painting area. We could, for instance, make sure that we only add points that are within our coordinate system constraints (0…kCanvasSize).

But for once, let’s go with the easiest way. Since our canvas will later ignore the points outside its drawing area for exporting an image, we don’t really care or bother that those points are there, but let’s get them out of the view.

Just wrap our CustomPaint widget with a ClipRect as follows:

child: ClipRect(
  child: CustomPaint(
    size: Size(kCanvasSize, kCanvasSize),
    painter: DrawingPainter(
      offsetPoints: points,
    ),
  ),
),

Decorating

If we want to get rid of that green background color, we probably should add some margins so we can see where the painting area is. For doing so, add this decorator to the Container that contains the Builder widget. Also remove the color property that it had. Should look like this:

Container(
  decoration: new BoxDecoration(
    border: new Border.all(
      width: 3.0,
      color: Colors.blue,
    ),
  ),
  child: Builder( 
    ...

If you want, you could add this values to the constants file if you wish, so all this properties are in one same file rather than spread across the app.

Also, we have gotten rid of the padding in the container, since we don’t need it at all.

After this last changes, we shouldn’t be able to paint outside the blue box and the app should look like this:

You made it again!

Congratulations! 👏 You ended up this tutorial with a pretty good state of the app we are trying to build. You really deserve more clapping from Steve Ballmer.

It was a long stretch but the good news is that the bulkiest part of the app is done!

I am aware that the content from this article is a bit heavier, we have been using several widgets that we probably haven’t used before and got loads of information about them. I really hope you understand the logic behind it and not so much how the widget works but what it is the purpose of the widgets at use.

I will be very happy to answer any question in the comments and I am expecting some feedback for the next article of this series.

In the next article we will dig into the brains of the app and how to get the image from our canvas and feed it to our model for a prediction. But you have all the tools in place for trying yourself before the next article, so if you feel adventurous, give it a go and compare your solution with the one coming in the next article.

As usual, you can access all the code from this section here.

sergiofraile/when_flutter_meets_tensorflow_part_3

Looking forward to see you in the next section! 👋

Flutter Community