HMS ML Kit — Live Image Classification

Hello everyone,

In this article, I will give some information about ML Kit Image Classification and talk about how you can develop the live image classification app. I will also share the sample codes for this integration.

What is ML Kit Image Classification?

The image classification service classifies elements in images into intuitive categories, such as people, objects, environments, activities, or artwork, to define image themes and application scenarios.

It supports both on-device and on-cloud recognition modes. On-device recognition refers to the process of running the detection algorithm model on the device and returning the recognition result. It supports more than 400 common image categories.

On-cloud recognition refers to the process of calling an API on the cloud where the detection algorithm model is running. It supports 12,000 image categories.

In addition, this service allows you to customize image classification models.

The image classification service is suitable for highly-functional apps that are capable of classifying and managing images with remarkable ease and precision. If you have a photo gallery app, integrating it with the image classification service provides for a new layer of intelligence, as images can be automatically sorted by category.

Compared with on-device recognition, on-cloud recognition is more accurate but less efficient and supports only static image recognition but does not support recognition from camera streams.

You can visit this link to learn more about HMS ML Kit Image Classification.

Image Classification

Development Steps

1. Integration

First, a developer account must be created and HMS Core must be integrated into the project to use HMS. You can access the article about that steps from the link below.

Android | Integrating Your Apps With Huawei HMS Core

2. Adding Dependencies

After HMS Core is integrated into the project and the ML Kit is activated through the console, the required library should be added to the build.gradle file in the app directory as follows.

dependencies {
    implementation "com.huawei.hms:ml-computer-vision-classification:$hmsImageClassificationVersion"
    implementation "com.huawei.hms:ml-computer-vision-image-classification-model:$hmsImageClassificationVersion"
}

Current Version : 3.2.0.300

Add the following statements to the AndroidManifest.xml file. After a user installs your app from HUAWEI AppGallery, the machine learning model is automatically updated to the user’s device.

<manifest
    ...
    <meta-data
        android:name="com.huawei.hms.ml.DEPENDENCY"
        android:value= "label"/>
    ...
</manifest>

If targetSdkVersion is 28 or later, add the following elements to manifest in AndroidManifest.xml to allow your app to download the model.

<manifest>
    ...
    <application    
        ...    
        android:usesCleartextTraffic="true" >    
        ...
    </application>
        ...
</manifest>

3.Adding Permissions

Add the following permissions to AndroidManifest.xml file.

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

4.Create Application Class

An Application class is required to launch the ML Kit when starting the application. ML Kit is launched in this class and it is provided to start with the project. Here, App Id must be given as parameter while initting ML Kit. After the BaseApplication class is created, it must be defined in the Manifest file.

5.Create Transactor Class

Create the ImageClassificationTransactor class for processing detection results. This class implements the MLAnalyzer.MLTransactor<T> API and uses the transactResult method in this API to obtain the detection results and implement specific services.

In this class, first the camera view should be set by calling the setOverlay() method. The parameters of the setOverlay() method must come from the View class. The next step will show how these parameters are set.

Then, create analyzeImageAndGetResult() method. Inside this method, the image classification result should be edited. The MLAnalyzer.Result<MLImageClassification?> object returns a list of all the results it detects in the image, along with the probability value. We have to parse the most probable result here and write it on the screen.

And finally, call the transactResult() method and get results. In this method, create a canvas object to print the result to the screen. Create a resultString object and get the most probable value by the analyzeImageAndGetResult() method. Next, set the properties required to print the result to the screen with the canvas.drawText() method. And, you will have the result printed on the screen.

The transactor class should be as follows.

6. Add Surface Views

Now we need to set the surfaceView values ​​to show the results on the camera and camera screen.

Firstly, create surfaceViewCamera and set layout parameters as “MATCH_PARENT”

Secondly, create surfaceViewOverlay and set layout parameters as “MATCH_PARENT” and set format as “TRANSPARENT”.

Then, create a TypeFace object to set your font family.

Once you’re done, call the setOverlay() method from the ImageClassificationTransactor class.

And finally, add your surfaceViews to your root layout and call the callback methods.

7. Prepare Lens Engine

In this step, instead of setting the LensEngine with a fixed value, using the device’s aspect ratio will give a healthier result. For this, we first need to check whether the screen is vertical or horizontal. After this control, you can adjust the lens settings as follows.

8. Create Analyzer

Now we have to set the ML Kit Image Classification values. The most important point here is that set the scene type to TYPE_ALL. Finally, set these settings on the analyzer object and call the transactor class.

9. Add CallBack

Finally, override the call back methods and start image classification.

Result

Tips & Tricks

• Don’t forget to set the App Id in the Application class.
• Make sure you have the necessary permissions before starting the analysis.
• If the Lens Engine settings are not correctly, there may be differences in the image.
• Always use the highest possible value as the default result.

References

Image Classification