Using the Coral USB accelerator to detect objects in images sent with MQTT
A little while back, I started building a raspberry pi camera security system, using openCV’s deep neural network module for object detection. (Thanks pyimagesearch for the great introductory resources!)
Unfortunately the pi was only able to handle around 0.8 frames of object detection per second using that approach. While my frame rate requirements are modest, that wasn’t going to cut it. Shortly after that I discovered the Coral USB accelerator, a low power, edge device that can do blazing fast tensor flow lite model inference, and its api is really simple to use!
Setup is extremely straightforward.
cd ~/
wget https://dl.google.com/coral/edgetpu_api/edgetpu_api_latest.tar.gz -O edgetpu_api.tar.gz --trust-server-names
tar xzf edgetpu_api.tar.gz
cd edgetpu_api
bash ./install.shYou’ll be asked about enabling maximum operating frequency. I selected no.
Below is a barebones example of using the coral for object detection, with the MobileNet SSD v2 quantized model.
First we specify the type of processing we want to do — image classification or object detection (in our case object detection) and pass it the pre-trained MobileNet model. After that, all thats left to do is prepare the image (Coral expects a PIL image) and call the DetectWithImage() method.
The detection results are a list of objects, each with a confidence score, label_id (used as a key to look up an associate object class), and bounding_box co-ordinates for the object.
The inference speed is limited by the raspberry PI 3’s USB 2.0 port data throughput. This morning, however, the Raspberry PI 4 was announced. One of its many cool features is the presence of USB 3.0 ports, which will provide huge improvements in inference speeds due to the greater data transfer rates. For now let’s get back to the pi3B+. I’ll update this when I get my hands on a Pi 4.
I’m using a camera resolution of 320x240 as my focus is on object detection, not a high definition video feed. At that resolution, I was able to achieve a frame rate of around 15 fps.
The camera is in night vision mode- in spite of what the image above might suggest, I do in fact have eyeballs.
To be honest, for a minimal viable product (MVP), I’m happy with detection frame rate of 5 fps for security purposes(being notified within 200 ms of someone entering a frame is definitely workable & since processing is happening locally, we’re already scoring by removing server roundtrip time, as well as reaping other benefits such as reduced bandwidth consumption, and data privacy.
With this in mind, and since I only have one coral accelerator at my disposal, I started working on a system that uses simple pi zero based cameras to ‘stream’ images to an edge processing hub which queues images and does object detection using the Coral, allowing multiple sources to leverage the hardware acceleration of the coral TPU (tensor processing unit). The images are sent using a standard IoT transport layer, MQTT.
Note: These are ongoing experiments aimed at finding a reasonable compromise between speed, simplicity of setup and cost, so I make no claims as to the overall efficiency of the image transport layer at this point. I’m at a, ‘does it work reasonably well’, phase in the development cycle.
Below are simplified scripts for the camera frame transmission over MQTT and subsequent object detection on the hub.
Cool, it works!
