Real-time object detection using new TensorFlow Lite Flutter Support

In my previous article, I announced the new TFLite Flutter Plugin and discussed building a Text Classification app.

However, for applications such as Image Classification or Object Detection writing the code for processing input and output manually could be cumbersome. Therefore, I am pleased to introduce the new TFLite Flutter Helper Library which can be used along with the plugin to build performant Flutter ML apps with ease.

The benchmarking results of the Flutter Plugin and Helper Library are close to that of official Java Support for native Android development. So, we can now build Flutter ML apps without compromising on performance and speed. ⚡️

In this article, We will discuss the usage of tflite_flutter_helper and build a truly realtime Object Detection Flutter App.

Getting Started

1. Create a new flutter project object_detection_app
2. Include tflite_flutterand tflite_flutter_helperin your pubspec.yaml.
3. Follow the initial setup instructions given here.
4. Download the model and place detect.tflite and labelmap.txt in the object_detection_app/assets/ directory.
5. Include assets/ in pubspec.yaml .

Let’s get started with code. 🚀

Converting CameraImage to Image

We will use the flutter camera plugin which captures a frame as CameraImage. We can convert this to Image using the code below.

Coding the classifier

Loading model and labels

Let’s create a class Classifier . We use Interpreter.fromAsset to create the interpreter and FileUtils.loadLabels to load labels as List<String> .

Pre-processing the input

We need to pre-process our raw image input to match the required dimensions of 300 X 300.

Create a TensorImage object and apply transformations using ImageProcessor.

Creating output buffers

The model outputs four float arrays as described here. We can create four TensorBufferFloat with corresponding shapes for storing output. The interpreter processes inputs and outputs in bytes format. TensorBuffer class eliminates the worry of manually inter-converting between double and their bytes representation.

// TensorBuffers for output tensors
TensorBuffer outputLocations = TensorBufferFloat(_outputShapes[0]);
TensorBuffer outputClasses = TensorBufferFloat(_outputShapes[1]);
TensorBuffer outputScores = TensorBufferFloat(_outputShapes[2]);
TensorBuffer numLocations = TensorBufferFloat(_outputShapes[3]);

Inference

When we are dealing with more than one input or output we can useinterpreter.runForMultipleInputs function provided by tflite_flutter.

The predict function will now look like this.

Now, the inference results will be stored in TensorBuffers.

Bounding Boxes from TensorBuffer

BoundingBoxUtils.convert for easily converting TensorBuffer to List<Rect>.

List<Rect> locations = BoundingBoxUtils.convert(
  tensor: outputLocations,
  valueIndex: [1, 0, 3, 2],
  boundingBoxAxis: 2,
  boundingBoxType: BoundingBoxType.BOUNDARIES,
  coordinateType: CoordinateType.RATIO,
  height: INPUT_SIZE,
  width: INPUT_SIZE,
);

Creating a list of recognitions

TensorBuffer.getDoubleValue ,TensorBuffer.getDoubleList , can be used to obtain the double values from the underlying ByteBuffer. Similarly for int as well.

// Prediction score
var score = outputScores.getDoubleValue(i);

// Label string
var labelIndex = outputClasses.getIntValue(i) + labelOffset;
var label = _labels.elementAt(labelIndex);

Inverse transform for bounding boxes

The bounding boxes in output correspond to the pre-processed input image of size 300 X 300 . We can obtain the boxes corresponding to the raw input image by applying inverse transform.

Rect transformedRect = imageProcessor.inverseTransformRect(
    locations[i], image.height, image.width);

Finally, our predict function will look like this,

And classifer.dart will look like this,

The Classifier is now ready and can be linked with a suitable UI.

You can view the complete app on Github. The code is well commented for better understanding.

Visit the repositories am15h/tflite_flutter_plugin, am15h/tflite_flutter_helper for more info.

Also, check out the Image Classification App.

Thanks for reading. ✌️