Firebase Series | Blog 8 | Firebase Object Detection and Labelling: Detect Objects and Categorize from Images Or videos.

One of the useful feature that firebase give to the developers is object detection. The implementation of this API is very similar to text extraction and face detection which I discussed in the previous blogs. The only difference of the API is that this includes both on-device object detection and cloud-based object detection. In this blog, I will be writing about the implementation of on-device object detection.

This API help developers scan the images or video frames and extract the list of objects present in the frame. But for the object detection on the device, there are few limitations. On-device object detection will detect the object and gives only a few set categories, unlike cloud-based object detection. Like shown below are the categories available in on-device detection.

Above mentioned categories are the only ones available for developers to use it for labelling objects when the on-device model is in use. But the good part is, almost all objects can be classified into any of these above categories. In my opinion, since we are doing an on-device classification, getting this much categories itself is a big deal. Waiting for code? Let’s jump in!!

1. Connect your Project with Firebase Face detection
2. Create a bitmap from the camera or the gallery. (In this project I am taking a picture using phone’s camera)
3. Using FirebaseVisionImage create to convert the bitmap to FirebaseVisionImage getting it ready to extract the text.
4. Create a FirebaseVisionObjectDetectorOptions with the features you need enabling in it.
5. Create an object of getOnDeviceObjectDetector and pass the bitmap into it and in the onSuccessListner work through the classification category to know what all objects in the category are present in the frame.

Connect your Project with Firebase Face detection

In your new project, select Tools>Firebase>

Select this option

Once connected you will see this screen.

Add the above meta tag inside the application tag in your manifest file. Once the above steps are done, understand that the Firebase is successfully connected to your project and all set for object detection and labelling.

Create a bitmap from the camera or from the gallery. (In this project I am taking a picture using phone’s camera)

The above code is very straight forward. takepicture method takes the user straight to the camera. Once the user clicks the picture and clicks on the “OK” button, the app will return the bitmap to the method activityforresult.

Using FirebaseVisionImage create to convert the bitmap to FirebaseVisionImage getting it ready to extract the text.

Now the FirebaseVisionImage is ready to pass it to detector.

Create a FirebaseVisionObjectDetectorOptions with the features you need enabling in it.

enableMultipleObjects() and enableClassification() is purely optional. Here we are enabling it to use the returned object for classification. We are using SINGLE_IMAGE_MODE to inform the detector that we are going to pass a single image to process it, feel free to try more IMAGE modes and explore more setting while creating options.

Create an object of getOnDeviceObjectDetector and pass the bitmap in to it and in the onSuccessListner work through the classification category to know what all objects in the category are present in the frame.

I have used all the categories available in the above code just to make your life easier to make you understand how many categories this on-device API gives to developers for classifying objects. To explore more, you can also check out the official documentation.

This is how the on-device model works for developers to detect objects from the images, now there is also cloud-based object detection and labelling which I will be discussing in the next blog. Meanwhile, you can get Firebase blaze package enabled in your account which we will be using it in cloud-based object detection and labelling.

You can find the whole project here in Github. This is a public repo which has many demo projects. You can use GitZip to download a specific project from this repo. Paste this link in GitZip and you have it.

Follow me on Twitter and medium. Feel free to contribute to Repo — AndroidAppSamples. Let’s grow together 💚