Firebase ML Kit 101 : Image Labeling
Image Labeling is the process of recognising different entities in an image.
You can recognise various entities like animals, plants, food, activities, colors, things, fictional characters, drinks etc with Image Labeling.
With ML Kit’s Image Labeling API, you can recognise 400+ entities (and 10,000+ with cloud-based image labeling).
Firebase ML Kit Series
In this series of articles, we will deep dive into different ML Kit APIs that it offers…
- Firebase ML Kit 101 : Introduction
- Firebase ML Kit 101 : Text Recognition
- Firebase ML Kit 101 : Face Detection
- Firebase ML Kit 101 : Barcode Scanning
- Firebase ML Kit 101 : Image Labeling (you’re here)
- Firebase ML Kit 101 : Landmark Recognition
- Firebase ML Kit 101 : Language Identification
- Firebase ML Kit 101 : Smart Reply
Let’s look into the ML Kit’s Image Labeling API and how we can integrate it into our apps.
ML Kit’s Image Labeling
ML Kit’s Image Labeling provides both on-device and cloud-based APIs.
You can choose which one to use depending on your use case.
The on-device API is fast and requires no internet connection. On the other hand, cloud based API is more accurate but requires an internet connection.
You can see the level of accuracy for both on-device and cloud-based APIs in the image below.
Note: Firebase ML Kit is in beta as of October ‘18.
Let’s Code!
Step 1 : Add Firebase to your app
Offcourse! You can add Firebase to your app by following the steps mentioned here.
Step 2 : Include the dependencies
You need to include the ML Kit dependencies in your app-level build.gradle
file.
dependencies {
// ...
implementation 'com.google.firebase:firebase-ml-vision:18.0.1'
implementation 'com.google.firebase:firebase-ml-vision-image-label-model:17.0.2'
}
Step 2.5 : Specify the ML models (optional)
For on-device APIs, you can configure your app to automatically download the ML models after it is installed from the Play Store. Otherwise, the model will be downloaded on the first time you run the on-device detector.
To enable this feature you need to specify your models in your app’s AndroidManifest.xml
file.
<application ...>
...
<meta-data
android:name="com.google.firebase.ml.vision.DEPENDENCIES"
android:value="label" />
<!-- To use multiple models: android:value="label,model2" -->
</application>
Step 3 : Get! — the Image
ML Kit provides an easy way to label images from variety of image types like Bitmap
, media.Image
, ByteBuffer
, byte[]
, or a file on the device. You just need to create a FirebaseVisionImage
object from the above mentioned image types and pass it to the model.
In my sample app I’ve used a Bitmap
image to create a FirebaseVisionImage
object.
val image = FirebaseVisionImage.fromBitmap(bitmap)
To create FirebaseVisionImage
object from other image types, please refer to the official documentation.
Step 4 : Set! — the Model
Now, It’s time to prepare our Image Labeling model.
ML Kit provides both on-device and cloud-based models for Image Labeling.
On Device Model
val detector = FirebaseVision.getInstance().visionLabelDetector
By default, the on-device image labeling API returns not more than 10 labels.
You can change this configuration by passing in an object of FirebaseVisionLabelDetectorOptions
to the image labeling model.
val options = FirebaseVisionLabelDetectorOptions.Builder()
.setConfidenceThreshold(0.8F)
.build()val detector = FirebaseVision.getInstance().getVisionLabelDetector(options)
Cloud Based Model
val detector = FirebaseVision.getInstance().visionCloudLabelDetector
By default, the cloud-based image labeling API uses the STABLE
version of the model and returns not more than 10 labels.
You can change this configuration by passing in an object of FirebaseVisionCloudDetectorOptions
to the image labeling model.
val options = FirebaseVisionCloudDetectorOptions.Builder()
.setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL)
.setMaxResults(15)
.build()
val detector = FirebaseVision.getInstance().getVisionCloudLabelDetector(options)
Step 5 : Gooo!
Finally, we can pass our image to the model for Image Labeling.
detector.detectInImage(image)
.addOnSuccessListener {
// Task completed successfully
}
.addOnFailureListener {
// Task failed with an exception
}
Step 6 : Extract the information
Voilà! That’s it!
If the image labeling was successful, the success listener will receive a list of FirebaseVisionLabel
objects. Each FirebaseVisionLabel
object represents the entity that was labeled and contains all the information related to it.
You can extract all this information like this.
Have a Look!
This is what you can achieve with ML Kit’s Image Labeling API.
Here’s the source code for the above app…
Firebase ML Kit Series
Don’t forget to have a look at other ML Kit APIs covered this series of articles.
- Firebase ML Kit 101 : Introduction
- Firebase ML Kit 101 : Text Recognition
- Firebase ML Kit 101 : Face Detection
- Firebase ML Kit 101 : Barcode Scanning
- Firebase ML Kit 101 : Image Labeling
- Firebase ML Kit 101 : Landmark Recognition
- Firebase ML Kit 101 : Language Identification
- Firebase ML Kit 101 : Smart Reply