Develop Your Image Classification App with a Trained Custom Model | ML Kit — Model Inference
Hi folks,
We continue our article series of ML Kit Custom Model with Model Inference feature. In the last post of the series, we will develop an image classification application by using a MindSpore model which is trained via HMS Toolkit. Let’s start by exploring what we learnt about ML Kit Custom Model to grasp the details better. If you have not read the previous articles yet, you can take a quick look before start. Here is the links:
Our aim will be developing an application which could classify different animal species and return accurate results for the inputs that provides by user by taking photos from the application or picking images from the gallery.
The model that we will use for our demo application has been trained with a data set containing around 1500–4000 photos for each animal species and categorizing 10 different species.
First of all, a labels.txt file is created and put in the assets folder together with the MindSpore model file with .ms extension in Android Studio project.
We must have completed the necessary development preparations for our project. You can find the project and app-level build.gradle files with required configurations.
Note: The dependencies to integrate HMS custom model SDK and MindSpore Lite inference framework have been added to app-level build.gradle file.
Note: Please pay attention to adding Maven repository in project-level build.gradle file.
We will use 2 different layouts in the project. The first one will be where we can take a photo by using the camera of the device or open the gallery to pick an existing image.
After photo is taken from the camera view or an image is picked from the gallery, another screen will be opened where we will show the image with the results of classification.
Let’s start by creating our first class which is LabelUtils where we will read the labels.txt file to fetch the categories of model by checking the assets folder via context. This process will be done within readLabels method and categories will be stored inside an ArrayList. Moreover, this object class will also include the processResult method where whole process will be completed by matching the label list with the probabilities that will return from our model. We will examine where we call these methods in the following steps.
In order to send a request to a model and to get the output that will return as a result of the request, it is necessary to specify input and output formats by complying the standards. We will create our second class which is called ModelOperator to determine input/output formats.
ModelOperator class will be extended in ImageLabelModel class where the methods of ModelOperator are overrided. The data types and shape of input/output will be defined within these methods. In addition, model name and path name will be set here.
Our next class will be InterpreterManager where we handle most of the works. A ModelOperator instance is created to get model name, path and to arrange settings of input and outputs. Since our animals.ms file is stored on the cloud, we will create an instance of MLCustomRemoteModel class and send a request to MLLocalModelManager class to download our model via download listener in asset method. In addition, a localModel object is created from MLCustomLocalModel class to ensure the existence of model in case of any problem that might be occurred during the model download process from cloud hosting. After the existence of model file is checked and modelExecutor instance is initialized executorImpl method will be called by passing the input bitmat as parameter.
We will trigger all these methods by picking an image from the device or by taking a photo, this image will be our input bitmap that we need to process in this step. Since the SDK supports 224*224 inputs, processBitmap method is used to crop all inputs to use the model by obeying this restriction.
After input instance is obtained via MLModelInputs class via processed bitmap and input/output settings to make an inference are set, exec method will be called through modelExecutor to send the request to our model with our input and settings.
The inference result will be obtained from the model and stored via SendArray class to get them in our activity class and to show all the results.
Until now, we have examined the stages to be applied after input has been provided. Now, we will see how to start this process in activity side. App will open with the camera view to take a photo and will have a button to pick image from gallery.
As you can see above, when user picks an image from MediaStore, bitmap is directed to InterpreterManager to start the process. If user directly takes a photo from camera view, interpreterManager instance will be created and used inside onCaptureSuccess method like below.
CameraXHelper class enables to use camera functions of the device in the app.
After model execution is completed in InterpreterManager, ClassificationActivity will be opened to show the input bitmap and classification results with the probabilities. When the bitmap and classification results are ready, they are stored via SendImage and SendArray classes. We will get and show them together on ClassificationActivity. ArrayList items will be monitored with a recycler view.
I think we made a good progress. Let’s see our demo application and how model inference works.
With this post, we have completed the article series of ML Kit Custom Model. I hope that they will assist you to realize your new ideas or existing projects with the help of Huawei Mobile Services.
Thank you for reading !
References
