Real-time object detection using new TensorFlow Lite Flutter Support
In my previous article, I announced the new TFLite Flutter Plugin and discussed building a Text Classification app.
However, for applications such as Image Classification or Object Detection writing the code for processing input and output manually could be cumbersome. Therefore, I am pleased to introduce the new TFLite Flutter Helper Library which can be used along with the plugin to build performant Flutter ML apps with ease.
The benchmarking results of the Flutter Plugin and Helper Library are close to that of official Java Support for native Android development. So, we can now build Flutter ML apps without compromising on performance and speed. ⚡️
In this article, We will discuss the usage of tflite_flutter_helper and build a truly realtime Object Detection Flutter App.
Getting Started
- Create a new flutter project
object_detection_app
- Include
tflite_flutter
andtflite_flutter_helper
in your pubspec.yaml. - Follow the initial setup instructions given here.
- Download the model and place
detect.tflite
andlabelmap.txt
in theobject_detection_app/assets/
directory. - Include
assets/
inpubspec.yaml
.
Let’s get started with code. 🚀
Converting CameraImage to Image
We will use the flutter camera plugin which captures a frame as CameraImage
. We can convert this to Image
using the code below.
Coding the classifier
Loading model and labels
Let’s create a class Classifier
. We use Interpreter.fromAsset
to create the interpreter and FileUtils.loadLabels
to load labels as List<String>
.
Pre-processing the input
We need to pre-process our raw image input to match the required dimensions of 300 X 300
.
Create a TensorImage
object and apply transformations using ImageProcessor
.
Creating output buffers
The model outputs four float arrays as described here. We can create four TensorBufferFloat
with corresponding shapes for storing output. The interpreter processes inputs and outputs in bytes format. TensorBuffer
class eliminates the worry of manually inter-converting between double and their bytes representation.
// TensorBuffers for output tensors
TensorBuffer outputLocations = TensorBufferFloat(_outputShapes[0]);
TensorBuffer outputClasses = TensorBufferFloat(_outputShapes[1]);
TensorBuffer outputScores = TensorBufferFloat(_outputShapes[2]);
TensorBuffer numLocations = TensorBufferFloat(_outputShapes[3]);
Inference
When we are dealing with more than one input or output we can useinterpreter.runForMultipleInputs
function provided by tflite_flutter.
The predict
function will now look like this.
Now, the inference results will be stored in TensorBuffers.
Bounding Boxes from TensorBuffer
BoundingBoxUtils.convert for easily converting TensorBuffer to List<Rect>.
List<Rect> locations = BoundingBoxUtils.convert(
tensor: outputLocations,
valueIndex: [1, 0, 3, 2],
boundingBoxAxis: 2,
boundingBoxType: BoundingBoxType.BOUNDARIES,
coordinateType: CoordinateType.RATIO,
height: INPUT_SIZE,
width: INPUT_SIZE,
);
Creating a list of recognitions
TensorBuffer.getDoubleValue
,TensorBuffer.getDoubleList
, can be used to obtain the double values from the underlying ByteBuffer. Similarly for int as well.
// Prediction score
var score = outputScores.getDoubleValue(i);
// Label string
var labelIndex = outputClasses.getIntValue(i) + labelOffset;
var label = _labels.elementAt(labelIndex);
Inverse transform for bounding boxes
The bounding boxes in output correspond to the pre-processed input image of size 300 X 300
. We can obtain the boxes corresponding to the raw input image by applying inverse transform.
Rect transformedRect = imageProcessor.inverseTransformRect(
locations[i], image.height, image.width);
Finally, our predict function will look like this,
And classifer.dart
will look like this,
The Classifier is now ready and can be linked with a suitable UI.
You can view the complete app on Github. The code is well commented for better understanding.
Visit the repositories am15h/tflite_flutter_plugin, am15h/tflite_flutter_helper for more info.
Also, check out the Image Classification App.
Thanks for reading. ✌️