Notes App with HUAWEI ML Kit

Introduction

Hello, developer! In this story I want to make a notes app from scratch with Ionic. We’ll cover multiple topics in our notes app. This application allows us to create, delete and edit notes. Save and list notes from storage and view all notes.

So where do we use ML Kit? We will add the Text Recognition, Text to Speech and Automatic Speech Recognition features of Huawei ML Kit to our application in order to speed up our note taking processes and to add the artificial intelligence structure to our application.

Less Talk, More Code 😎

Text Recognition with ML Kit

The text recognition service can extract text from images of receipts, business cards, and documents. This service is useful for industries such as printing, education, and logistics. You can use it to create apps that handle data entry and check tasks.

In our application, first we extract text from our photos and we will add this text to note.

Before the creating app, you must register as a HUAWEI developer and complete identity verification on HUAWEI Developers. After the login, we should create an AppGallery Project.
1. Sign in to AppGallery Connect, and click My projects.
2. Click Add project.
3. Enter a project name and click OK.

4. After the create project, go to Project Settings>General Information section and click Add app.
5. On the Add app page, enter the app information.

6. On the Project settings page, enter SHA-256 certificate fingerprint.
7. For the enable game service API go to Project Setting> Manage APIs enable ML Kit.

8. Download the configuration file agconnect-services.json for Android platform.

Create an App

Before the start demo application, make sure you have installed Node.js, npm and ionic CLI.

1. First of all, we will generate a Ionic application.

ionic start ionic-angular-notesapp blank — type=angular

2. Go into the project directory.

3. Don’t forget to update the widget id property which is specified in the config.xml file. It must be same with client > package_name value of the agconnect-services.json file

4. Enable the Cordova integration.

ionic integrations enable cordova

5. Add the Android platform to the project.

ionic cordova platform add android

6. Install `HMS ML plugin` to the project.

ionic cordova plugin add @hmscore/cordova-plugin-hms-ml
npm install @ionic-native/core @hmscore/ionic-native-hms-ml

7. Copy agconnect-services.json file to <project_root>/platforms/android/app directory.

8. Add keystore(.jks) and build.json files to your project’s root directory.

9. Build the app.

ionic build

10. Run the app.

ionic cordova run android — device

If everything is okay, this screen will be appear

All settings are ready. We will be one step closer to completing our application. Let’s add a page and service.

ionic g page Detail
ionic g service services/Notes

In the app-routing.module.ts, I change detail page’s routing and home page name to ‘notes’.

{
    path: ‘’,
    redirectTo: ‘notes’,
    pathMatch: ‘full’},
{
    path: ‘notes/:id’,
    loadChildren: () => import(‘./detail/detail.module’)
                   .then( m => m.DetailPageModule)
},

I have added the process of adding and deleting notes in the simplest way, you can see the details on Github.

For using ML Kit services, we need to set the Api Key first. Copy the Api Key from agconnect-services.json call the HMSMLKit.serviceInitializer service.
I called this services with the ionViewWillEnter lifecycle method.

We have to select the notes photos from the gallery on our phone, we need to give the necessary permissions to our application, I called HMSMLKit.requestPermissions function for it. Thus, we will benefit from all the services of ML Kit and we have all permissions.
Add the code below to extract text from the photo that we will choose from the gallery.

The first issue we need to pay attention to here is the stage of selecting photos from the gallery with the FileChooser plugin. We should call HMSTextServiceProvider.imageTextAnalyser method after the choosing process. If the file selection is not successful or URL is not correct, ML Kit will send an error message. You need to be careful because of the file selection and calling ML Kit processes are asynchronous.

Let’s make it little spicy 😋

Automatic Speech Recognition with ML Kit

Automatic speech recognition (ASR) can recognize speech not longer than 60s and convert the input speech into text in real time. This service uses industry-leading deep learning technologies to achieve a recognition accuracy of over 95%. Currently, Mandarin Chinese, English, French, German, Spanish, Italian, and bilingual speech (Chinese-English only) can be recognized.

While our notes application had the ability to extract text from pictures, it would have been possible that it could not have a voice note feature.
We will send an alert message to the user with the help of a button and add the ASR feature to the code.

Text to Speech with ML Kit

Text to speech (TTS) can convert text information into audio output in real time. Rich timbres are provided and the volume and speed can be adjusted, thereby natural voices can be produced. This service uses the deep neural network (DNN) synthesis mode and can be quickly integrated through the on-device SDK to generate audio data in real time.

It could be easier if there was someone reading our notes. ML Kit provides us this feature with the TTS module.

We can read the note that appears on the detail page very easily with the TTS module.

Tips & Tricks

1. If the API key is not set, we cannot benefit from most features.
2. ML Kit supports multiple languages, you can change your application language easily.

Conclusion

In this article, we made a simple application more comprehensive. You can use Huawei ML Kit to support your ideas.
For example, you can add simple translation methods to this note application. You can always take a step further.
For more details, you can visit our developer page.

References

HMS-Core/hms-cordova-plugin

Document

Mervetafrali/Notes-App-with-Ml