Implementing Smarter Android Apps with ML Kit

Photo by Kayvan Kaseb

Nowadays, a growing interest to Machine Learning forced the companies to focus and invest on it. Machine Learning (ML) is the major of computational science, which focuses on analyzing and interpreting patterns and structures in data to enable learning, reasoning, and decision making outside of human being interaction. Thus, the introduction of new mobile SDK called ML Kit has been a prominent topic in the IT industry. This essay aims to discuss ML Kit as a new SDK for having smarter Android apps, and implement a sample Android project via ML Kit for face detection as well.

Overview

Nowadays, the modern mobile phones have enough capacity to process as much as traditional computers can, and our phones and tablets are powerful enough to run applications, which can learn and interact in real time. Machine Learning(ML) is helping us build smart apps. For instance, the Google Apps prediction for the next apps you are likely to use, and a mobile app, which solves the world’s biggest problem for finding the perfect emoji. Besides, Smart Assistants, Snapchat Filters, Smart Reply, and Google lens are some of the other products of utilizing Machine Learning on devices. A number of SDKs exist on mobile phones that help run Machine Learning on mobile devices. Some of them includes software development kits developed by popular companies such as Amazon, Intel, and Microsoft, and also Google has developed the ML Kit. The ML Kit enables Android developers to build smarter apps by utilizing Machine Learning.

What is the definition of Machine Learning (ML)?

Machine learning (ML) is the study of computer algorithms that enhance automatically through experience. It is seen as a subset of Artificial Intelligence (AI). Machine learning algorithms build a mathematical model based on sample data, known as “training data”, in order to make predictions or decisions without being explicitly programmed to do.

In fact, we use algorithms to learn from data and then we make predictions about similar data that has not been seen before by using Machine Learning. Thus, it would be a two-step process: training and inference. First, the model learns, and then we use it to make predictions. The process of model learning is what we typically call training, and when the model is making predictions about the data is what we call inference. This is a high-level view of what is happening during training the model for extracting knowledge from input data with using prediction and decision. Because we know what the right answer in our case, we are able to calculate the errors that occurs many times. As a result, we use these errors to enhance our model, and this process is repeated many times until we reach the point that we make sure that the model is good enough or that this is the best result that we can do. This involves a number of steps and preparations and that is why we have to use a framework to make this easier.

ML Kit for Android developers

Basically, ML Kit SDK is a new product from Google, which was presented in 2018. ML Kit is a software development kit that makes it possible for developers to simplify the integration of machine learning models into their mobile apps. In addition, even a junior developer can address this task easily.

A new SDK that brings Google’s Machine Learning experience to mobile developers in a powerful, easy-to-use package on Firebase.

ML Kit has some features that can be categorized in three sections:

1. Vision: Video and image analysis APIs to label images and detect bar-codes, text, faces, and objects.
2. Language: Natural language processing APIs to identify and translate between 58 languages and provide reply suggestions.
3. Custom: Build your own models using AutoML Vision Edge, or bring your own TensorFlow Lite models, run experiments, and deploy them in production with ML Kit.

Each category has some features. For example, Vision has some features as Base APIs: Barcode Scanning, Face Detection, Image Labeling, Object Detection, Landmark Detection, and Text Detection.

Benefits of utilizing ML Kit in Android apps

1. Easy-to-use framework for developers

Typically, adding ML to an Android app is a challenging and time-consuming process. By using this new SDK, this process is dramatically simplified. All you need is to pass data to the API, and wait until SDK will send a response. Google’s team stated, that implementing their APIs. Therefore, you don not need deep knowledge of neural networks.

2. Custom models

This option is helpful for experienced developers. If base ML Kit APIs do not cover all your needs, you can have your own ML model. New SDK can work with Tensor Flow and Android machine learning library. Besides, it supports mobile developers with the possibility to download their own model to Firebase console, and bundle it with their products. Another important key factor is that models are updated dynamically. It suggests that model will be updated even without updating the whole app.

3. Cloud and on-device APIs

In fact, developers can choose between cloud-based and on-device APIs. To make a right choice it is significant to take into consideration some differences between these two variants. Cloud APIs process data on the Google Cloud Platform; therefore, recognizes objects more accurately. However, cloud models are larger in comparison to on-device ones. Offline models need less free space, can work offline and process data faster, but their accuracy is lower.

4. Security Concern and privacy

As we know, most models are app-specific. So, probably, your customize model should be protected from other accesses from other developers. Now, you can do this task safely by using ML Kit. Also, Google claims that cloud APIs do not store user’s data, and it is removed when processing is done.

In addition to this section, you can make your model smaller and it runs faster on mobile devices by using conversion and compression. Also, A/B testing and report performance can be done via Analytics for A/B testing.

Face Detection

There are a lot of possibilities in you apps where you may want to utilize the detection of faces. Probably, you want to perform facial verification, tag photos, or add filters to a camera. Now, this feature is available as a part of the ML Kit. When API gets an image, it returns coordinates of nose, mouth, eyes, cheeks, and ears. Furthermore, this API supports the information whether a person on the picture has eyes closed or smiles. However, take into consideration that this determination works only for frontal faces. Another important aspect is the speed of face detection. As API performs tasks extremely immediately, it can be used in real-time apps.

Also, it is important to note that face detection via ML Kit is just only available as on-device recognition. So, you cannot perform face recognition on the cloud. This is probably a noticeable point for your users in terms of their personal data and privacy.

By using ML Kit’s face detection API, you can detect faces in an image, identify key facial features, and get the contours of detected faces. With face detection, you can get the information you need to perform tasks like decorating selfies and portraits or generating avatars from a user’s photo. Since ML Kit can accomplish face detection in real time, you can use it in applications like video chat or games that respond to the player’s expressions.

Picture from Face Detection ML Kit Tutorial by Google

To implement Face Detection via ML Kit in Android, there are three main steps as follows:

1. Configure the Detector Options
2. Run the Detector
3. Retrieve the Information

Now, these steps are mentioned with more information as follows:

1. Configure the Detector Options

The first step to do for both on-device and cloud is to expose functionality. The best APIs expose functionality to you with detectors both for on-device and cloud. This can be configured on all the options the API provides to give you more specific results.

Performance mode: To select speed or accuracy when detecting faces.

Detect landmarks: To identify facial landmarks such as eyes, ears, nose, cheeks, mouth, and so on.

Detect contours: To detect the contours of facial features. Contours are detected for only the most prominent face in an image.

Classify faces: to classify faces into categories such as “smiling”, and “eyes open”.

Minimum face size: float default: 0.1f

Enable face tracking: To assign faces an ID, which can be used to track faces across images.

val detectorOptions = FirebaseVisionFaceDetectorOptions.Builder()
        .setPerformanceMode(ACCURATE)
        .setLandmarkMode(ALL_LANDMARKS)
        .setClassificationMode(ALL_CLASSIFICATIONS)
        .setContourMode(ALL_CONTOURS)
        .setMinFaceSize(0.1F)
        .enableTracking()        
        .build()

2. Run the Detector

The options are configured in the previous section. This step is where we should use them. First of all, you should get an instance of the FirebaseVision, and then add the detectorOptions to it.

detector = FirebaseVision.getInstance().getVisionFaceDetector(detectorOptions)

Then, create a FirebaseVisionImageMetadata object that contains the image's height, width, color encoding format, and rotation.

metadata = FirebaseVisionImageMetadata.Builder()
    .setWidth(cameraWidth)
    .setHeight(cameraHeight)
    .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
    .setRotation(FirebaseVisionImageMetadata.ROTATION_270)
    .build()

To detect faces in an image, create a FirebaseVisionImage object from either a Bitmap, media.Image, ByteBuffer, byte array, or a file on the device. Then, pass the FirebaseVisionImage object to the FirebaseVisionFaceDetector's detectInImage method.

private fun detect(firebaseVisionImage: FirebaseVisionImage) {
    detector.detectInImage(firebaseVisionImage)
        .addOnSuccessListener(successListener)
        .addOnFailureListener(failureListener)
}

Eventually, you can implement these two callbacks for making sure about completing the tasks.

3. Retrieve the Information

In the last step you can be able to retrieve a collection of faces, then we can parse through each face, and then perform whatever actions we want on it. For instance, the degree of the happiness is calculated in this Android app as below:

for (face in faces) {
if (face.smilingProbability !=    FirebaseVisionFace.UNCOMPUTED_PROBABILITY) {
                val smileProb = face.smilingProbability
                if (smileProb > 0) {
                    Toast.makeText(this, "The degree of your happiness is:$smileProb", Toast.LENGTH_SHORT).show()
                }
            }
        }

For more information and detail about implementation, you can observe the sample code on the Github:

kayvankaseb/facedetection

In conclusion, recently. a growing interest to Machine Learning forced the companies to focus and invest on it. Therefore, the introduction of new mobile SDK called ML Kit has been a prominent topic in the IT industry. Although ML Kit has some prominent aspects particularly for Android developers that are mentioned in this article, there are some negative issues are remained to address like the size of custom model.

Understand Machine Learning