Detect Ingredients from Food Image: Combine the Forces of Huawei ML Kit and Image Kit

This week, I was asked to build an application that can detect base elements of food given in an image. This article will tell you about my journey building this application using Huawei Image Kit and Machine Learning Kit.

Huawei ML Kit provides many useful machine learning related features to developers and one of them is Image Classification. Thanks to this feature, ML Kit can describe images by classifying their elements. However, if you need to get the names of the elements that appear in an image, ML Kit may not provide the data you want efficiently. I will now show you how I tried to overcome this issue.

First of all, I started by integrating HMS Core into the application which is explained in detail here:

Android | Integrating Your Apps With Huawei HMS Core

Machine Learning Kit Part

After the HMS Core integration, I created a method which takes a bitmap and returns classification of that. Thanks to Huawei ML Kit, it can be done in few lines of code:

When I tried this method, I observed that whenever I send an image of a plate of food into it, ML Kit returns classifications like “Food, Dish, Meal, Lunch” instead of the actual elements of the food.

At this point, I understood that ML Kit wants to describe pictures in a general way, so it doesn’t always return the elements appearing in pictures.

Let’s see an example:

Before Image Kit

As you can see, even though the picture contains egg and salad, they don’t appear among the classifications.

Knowing that ML Kit has the power to detect these elements, I came up with an idea that will force ML Kit not to ignore them. My idea was cropping the image into small pieces and send them to ML Kit separately. I thought that ML Kit will be forced to return elements in images with this way since there won’t be other elements to get things complicated.

Image Kit Part

At this moment, a need for help arose to crop the images in the background. That is when I decided to bring Image Kit into the field. Image Kit provides a layout element called “CropLayoutView” which helps users to select a portion of image to be cropped easily. However, I didn’t want users to interact with this view because my aim was to crop the image in the background without user even knowing that.

Therefore, I decided to do a little trick by making this element invisible:

With this way, cropping process can be done in the background without showing anything in UI.

Then, I wrote a method which takes a bitmap and returns a list of bitmaps consists of cropped pieces of the main image:

Let me explain what is going on here. First, we need the size of the image to decide the amount we crop. Hence, we get height and width of the bitmap. Then, we arrange how we want to crop the picture by using the height and the width. The cropImage method simply crops the given image by given points with the help of CropLayoutView of Image Kit.

Note that, when height and width are both equal to zero, it points the top left of the image. As the height goes to its value from zero, we are going towards the bottom of the image and as the width goes to its value from zero, we are going towards the right side of the image. You should do your calculations according to that when deciding on cropping points.

Here, I used 6 different images to be sent to ML Kit which are:

• Uncropped, original version of the image
• A piece from middle of the image
• A piece from right side of the image
• A piece from left side of the image
• A piece from top side of the image
• A piece from bottom side of the image

Conclusion

I used Image Kit to crop any image into pieces before sending it to ML Kit for image classification. Then, I sent these pieces to ML Kit separately and collected the results of each. I observed that this approach increased the number of elements detected dramatically.

Thanks to this approach, ML Kit now returns both egg and salad for the example picture which was failed earlier:

After Image Kit

References

I have shown you the key points of the application but you can find the source code for the complete version on GitHub:

bugra7/Xecipes

Note that, you will not be able to run this application because you don’t have the agconnect-services.json file for it. Therefore, you should only take it as a reference to create your own application.

And lastly, if you have any questions, you are always welcome to ask them on HUAWEI Developer Forum:

HUAWEI Developer Forum - HUAWEI Developer