Real-time Object Detection in Flutter

4 min readFeb 18, 2019

Recently Flutter team added image streaming capability in the camera plugin. This allows you to capture the frame in a live camera preview.

I created a demo app that uses image streaming with tflite (TensorFlow Lite) plugin to achieve real-time object detection in Flutter.

Here’s a short video captured on my iPad demonstrating the app.

Image streaming in camera plugin

Refer to the link to add the camera plugin to the Flutter project.

To start image streaming, call startImageStream in the camera controller. The method is triggered every time a new frame arrives.

controller.startImageStream((CameraImage img) { <YOUR CODE> });

The output CameraImage class has 4 members: image format, height, width and finally planes which consists of the bytes of the image.

class CameraImage {
  final ImageFormat format;
  final int height;
  final int width;
  final List<Plane> planes;
}

The format of the image varies with the platforms:

Android: android.graphics.ImageFormat.YUV_420_888
iOS: kCVPixelFormatType_32BGRA (Note that the format was kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange in v2.8.0. Later in v4.0.0 it was changed back to 32BGRA.)

As a result of the different format, the output CameraImage on iOS and Android are different:

Android: planes is a list of bytes arrays of Y, U and V planes of the image.
iOS: planes contains a single array containing the RGBA bytes of the image.

Knowing the format is important for properly decoding the image and feeding it to TensorFlow Lite.

Decoding CameraImage

It’s possible to decode the image in Dart code but unfortunately the image library works slowly on iOS. Decoding the frames from native code is faster and more efficient.

iOS:

Since the format is already RGBA, we only need to extract the Red, Green, Blue values from the bytes and feed them to the input tensor of a TensorFlow Interpreter.

Android:

We need to first create a bitmap in RGBA format from the bytes of YUV planes. An easy way is to use a render script to achieve the conversion.

The code that converts NV21 to RGBA is as below. I got it from https://stackoverflow.com/a/36409748.

Once we have the raw bitmap, we can resize it to fit the input size required and feed the RGB values to the input tensor.

Detect objects using tflite plugin

The tflite plugin wraps TensorFlow Lite API for iOS and Android. It implemented native code for feeding input and extracting output of popular models. For object detection, it supports SSD MobileNet and YOLOv2.

The plugin provides a detectObjectOnFrame method which can decode image stream from camera plugin (under the hood it uses the code described above), run inference and return the recognitions.

We can simply pass the planes bytes of CameraImage to the method and get the detected objects.

The output is a list of objects in the following format:

{
  detectedClass: “hot dog”,
  confidenceInClass: 0.123,
  rect: {
    x: 0.15,
    y: 0.33,
    w: 0.80,
    h: 0.27
  }
}

x, y, w, h define the left, top, width and height of the box that contains the object. The values are between [0, 1]. We can scale x, w by the width and y, h by the height of the image.

Display the outputs

A small problem with camera plugin is the preview size does not always fit the screen size. It’s recommended to put the preview in an AspectRatio widget to avoid stretching it but that will leave margins.

I personally prefer the camera preview to take the whole screen so I find a workaround by putting the preview in an OverflowBox widget: first compare the height/width ratio of the preview and the screen, then scale the preview to cover the screen by fitting either the screen width or screen height.

Accordingly, when drawing the boxes, scale the x, y, w, h by the scaled width and height. Note that x or y has to be subtracted by the difference between scaled height(width) and screen height(width), since a part of the preview is hidden behind OverflowBox.