Machine Learning in Flutter

Machine Learning and Flutter are two of my favorite topics in which I like to explore more and more! In this article, I’ll be explaining how you can incorporate Machine Learning into your flutter app using the power of plugins in flutter. So let’s get started!

I’ll be explaining how to make an app with multiple ML features in it with the help of firebase_ml_vision, tflite and an API built by me!

Features that we’ll be making:

• Optical Character Recognition along with conversion of the recognized text into speech.
• Face Detection.
• Object Detection.
• Image Labeling along with conversion of the Image Labels into speech.
• Classify names into male or female, using the API developed by me in my project, Genderify.

Home Page

Getting Started:

• Dependencies:

• Add Firebase to your project:

Using the Firebase Console add an Android app to your project: Follow the assistant, download the generated google-services.json file and place it inside android/app. Next, modify the android/build.gradle file and the android/app/build.gradle file to add the Google services plugin as described by the Firebase assistant.

For more clarity you can refer to the video. https://www.youtube.com/watch?v=DqJ_KjFzL9I

• Insure that the minimumSdkVersion is at least 21 in the defaultConfig block under android block in the android/app/build.gradle file. If not, change it.

1. Optical Character Recognition

• Pick Image using Image Picker

The source of the image can be changed to camera.

The Image is decoded because that is the requirement of the OCR in order to read the text.

• Reading the text from the image and feeding the recognized text to the Text To Speech(TTS) library.

OCR Demonstration. Please ignore the “tomorrow” hidden by the floating action button and the small font size.

2. Face Recognition

• Pick Image as shown before.
• Create an Instance of Face Detector and call processImage with visionImage.
• Add the coordinates of the faces to a <Rect> type List.
• These coordinates will be used by the Custom Painter to draw the bounding boxes on the image.

Call the Custom Painter class and pass to it the <Rect> List and the image(I will link my github repo at the end of this article, so you can see how I’ve implemented everything).

Face Detection Demonstration. Two of my favorite players in one frame.

3. Image Labeling

Add the following in your android/app/build.gradle file.

android {
    dependencies {
        // ...

        api 'com.google.firebase:firebase-ml-vision-image-label-model:17.0.2'
    }
}

Coming this far, you know the first step is always to pick the image.

• Use the Image Labeler class to detect all Labels in the selected image.
• Feed the label text to TTS object in order to convert it into speech.

Image Labeler Demonstration. These are some of my college friends.

4. Object Detection

This part is the most unstable part of my project but I’m working on it. The Object Detecting library poorly detects objects and most of the time gives bad results. I won’t be attaching github gists in this one as it would make this portion a bit lengthy so you can refer to the source code which I’ve attached at the end of this article.

Now, for this you need to have a preloaded tflite and yolo model in your project. Make a new assets folder and store the models in it. You can refer to the assets folder in my repository to download the models from there.

Also, you need to add the following in android/app/build.gradle in android block.

aaptOptions {
        noCompress 'tflite'
        noCompress 'lite'
    }

Now everybody knows how to pick an Image so let’s skip that.

• Load the tflite model inside initState() .
• Apply predictions on the selected image.
• Render the boxes to make them visible on the image along with the tag and confidence level.

Object Detection Demonstration.

Bonus part :

5. Genderify : Classifying names into male and female based on gender.

Well this doesn’t use a flutter plugin but an API built by me in my project:

mayanktolani19/Genderify

It is hosted on heroku. Here’s how you access the API.

The model is trained on foreign names so it gives less accuracy on Indian names.

Change the last parameter in the url to your desired name.

Genderify Demonstration.

Here’s the source code to my complete project. Please do leave a star. I’m welcoming UI contributions, if you want to!

mayanktolani19/flutter_mlkit

This was my first medium article, worked hard on this, so please leave some claps if you liked my work. That would encourage me to write more in the future!