Firebase ML Kit 101 : Image Labeling

Image Labeling is the process of recognising different entities in an image.

You can recognise various entities like animals, plants, food, activities, colors, things, fictional characters, drinks etc with Image Labeling.

With ML Kit’s Image Labeling API, you can recognise 400+ entities (and 10,000+ with cloud-based image labeling).

Firebase ML Kit Series

In this series of articles, we will deep dive into different ML Kit APIs that it offers…

• Firebase ML Kit 101 : Introduction
• Firebase ML Kit 101 : Text Recognition
• Firebase ML Kit 101 : Face Detection
• Firebase ML Kit 101 : Barcode Scanning
• Firebase ML Kit 101 : Image Labeling (you’re here)
• Firebase ML Kit 101 : Landmark Recognition
• Firebase ML Kit 101 : Language Identification
• Firebase ML Kit 101 : Smart Reply

Let’s look into the ML Kit’s Image Labeling API and how we can integrate it into our apps.

ML Kit’s Image Labeling

ML Kit’s Image Labeling provides both on-device and cloud-based APIs.
You can choose which one to use depending on your use case.

ML Kit’s Image Labeling API available both on device and in cloud

The on-device API is fast and requires no internet connection. On the other hand, cloud based API is more accurate but requires an internet connection.

You can see the level of accuracy for both on-device and cloud-based APIs in the image below.

image courtesy: Google I/O ’18

Note: Firebase ML Kit is in beta as of October ‘18.

Let’s Code!

Step 1 : Add Firebase to your app

Offcourse! You can add Firebase to your app by following the steps mentioned here.

Step 2 : Include the dependencies

You need to include the ML Kit dependencies in your app-level build.gradle file.

dependencies {
  // ...

  implementation 'com.google.firebase:firebase-ml-vision:18.0.1'
  implementation 'com.google.firebase:firebase-ml-vision-image-label-model:17.0.2'
}

Step 2.5 : Specify the ML models (optional)

For on-device APIs, you can configure your app to automatically download the ML models after it is installed from the Play Store. Otherwise, the model will be downloaded on the first time you run the on-device detector.

To enable this feature you need to specify your models in your app’s AndroidManifest.xml file.

<application ...>
  ...
  <meta-data
      android:name="com.google.firebase.ml.vision.DEPENDENCIES"
      android:value="label" />
  <!-- To use multiple models: android:value="label,model2" -->
</application>

Step 3 : Get! — the Image

ML Kit provides an easy way to label images from variety of image types like Bitmap, media.Image, ByteBuffer, byte[], or a file on the device. You just need to create a FirebaseVisionImage object from the above mentioned image types and pass it to the model.

creating FirebaseVisionImage object from different image types

In my sample app I’ve used a Bitmap image to create a FirebaseVisionImage object.

val image = FirebaseVisionImage.fromBitmap(bitmap)

To create FirebaseVisionImage object from other image types, please refer to the official documentation.

Step 4 : Set! — the Model

Now, It’s time to prepare our Image Labeling model.
ML Kit provides both on-device and cloud-based models for Image Labeling.

On Device Model

val detector = FirebaseVision.getInstance().visionLabelDetector

By default, the on-device image labeling API returns not more than 10 labels.

You can change this configuration by passing in an object of FirebaseVisionLabelDetectorOptions to the image labeling model.

val options = FirebaseVisionLabelDetectorOptions.Builder()
        .setConfidenceThreshold(0.8F)
        .build()
val detector = FirebaseVision.getInstance().getVisionLabelDetector(options)

Cloud Based Model

val detector = FirebaseVision.getInstance().visionCloudLabelDetector

By default, the cloud-based image labeling API uses the STABLE version of the model and returns not more than 10 labels.

You can change this configuration by passing in an object of FirebaseVisionCloudDetectorOptions to the image labeling model.

val options = FirebaseVisionCloudDetectorOptions.Builder()
      .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL)
      .setMaxResults(15)
      .build()

val detector = FirebaseVision.getInstance().getVisionCloudLabelDetector(options)

Step 5 : Gooo!

Finally, we can pass our image to the model for Image Labeling.

detector.detectInImage(image)
        .addOnSuccessListener {
            // Task completed successfully
        }
        .addOnFailureListener {
            // Task failed with an exception
        }

Step 6 : Extract the information

Voilà! That’s it!
If the image labeling was successful, the success listener will receive a list of FirebaseVisionLabel objects. Each FirebaseVisionLabel object represents the entity that was labeled and contains all the information related to it.

You can extract all this information like this.

Have a Look!

This is what you can achieve with ML Kit’s Image Labeling API.

Here’s the source code for the above app…

hitanshu-dhawan/FirebaseMLKit

Firebase ML Kit Series

Don’t forget to have a look at other ML Kit APIs covered this series of articles.

• Firebase ML Kit 101 : Introduction
• Firebase ML Kit 101 : Text Recognition
• Firebase ML Kit 101 : Face Detection
• Firebase ML Kit 101 : Barcode Scanning
• Firebase ML Kit 101 : Image Labeling
• Firebase ML Kit 101 : Landmark Recognition
• Firebase ML Kit 101 : Language Identification
• Firebase ML Kit 101 : Smart Reply

The full source-code with other ML Kit APIs can be found here!

hitanshu-dhawan/FirebaseMLKit

Thanks for reading! Share this article if you found it useful.
Please do Clap 👏 to show some love :)

Let’s become friends on LinkedIn, GitHub, Facebook, Twitter.