Piano Chords Prediction With AI (Teachable Machine)
Recognising piano chords with AI in only two lines of code sounds impossible? Check this:
Done…easy right?
If you would like to learn how to incorporate this code into your project, stick around! We will create a hybrid app that your friends and family can use to learn piano chords!
Link for the whole repository can be found here.
Terminology
Let’s look at the technologies applied in this project.
Teachable Machine (TM)
A web-based tool which simplifies the process of training the machine learning model with easy-to-use GUI. As of writing this article, TM provides three classifying models — for images, sounds and poses. We will use a sound classification model to recognise four different piano chords.
ml5.js
Open source high level interface built on top of Tensorflow.js for building ML applications. It aims to improve public understanding of machine learning by making ML approachable for a broad audience of artists, creative coders, and students.
Monaca
“Cross-platform hybrid mobile app development platform and tools in the cloud.” In this app, Monaca will serve as an easy solution for deployment on a mobile device. You also have the possibility of development in the cloud with Monaca Cloud IDE, so you do not have to download anything on your computer.
Prerequisites:
- Node v14.19.0 or higher,
- free account on Monaca,
- piano (online version is enough).
Workflow:
- Train the models with Teachable Machine — for better results, we will train two models (one for laptop and one for mobile).
- Use ml5.js to load the model and start predictions.
- Create a GUI to show feedback for user.
- Deploy the application for phone.
App Wireframe
Training The Model
As a first step, we need to train the model. Open Teachable Machine website and select Audio Project.
You will be presented with GUI that walks you through the process of training the model. There are two default classes: Background noise and Class 2. For the simplicity of this demo, we will use only four chords, namely C, D, E and F (feel free to add more), therefore we need four classes for chords + one class for background noise (this class is always present, so that the model knows when there is no sound happening — you can record silence in the room where your piano is located).
As for the number of samples, it is better to have more than the recommended minimum — around 40 samples can already make the predictions accurate. Since we want to make hybrid application, we need to train two independent models. This is due to different quality of microphone on phone and laptop, that might affect the accuracy of predictions.
There is one problem which occurs if you want to upload audio recorded by phone microphone. The Teachable Machine does not work on mobile devices, nor you can upload your own audio (it must be directly recorded through TM website). To tackle this issue, download EZ-mic or similar software, which will allow you to use phone microphone as an external laptop microphone.
After training the model, you should see similar view as below — when pressing individual chords, the output shows prediction in percentages.
The training part is done, now we can do some coding.
Setting up The Project
To simplify the development, we can start from a template provided by Monaca CLI. After creating an account, open the console and write the following commands:
$ npm install -g monaca // install Monaca CLI
$ monaca login // login with your new account
$ monaca create AI-Piano-App // create new projectChoose: Onsen UI and Vue.js -> Framework7 Vue Blank
After this, you should see a new folder “AI-Piano-App” created. Open it in your preferred code editor.
The UI will consist of two Vue components. Create “LoadingPage.vue” and “PredictingPage.vue” in “./pages” directory. The first mentioned page serves as an initial page displayed for the user while the model is being loaded. It consists only of HTML code, so you can copy it from the repository. “PredictingPage.vue” displays the chord to play. If the user does not know the chord, he/she can tap the card and flip it to see the chord’s picture. After successful play and recognition of it, the card turns green and a new chord is displayed. Here is the Vue code which uses Composition API:
The component starts with variable and function definitions, after that, it uses onMounted lifecycle hook to set up the component. On line 19 a function from a different module is called to start the prediction. Let’s implement it.
Create ”helperFunctions,js” in “./js” directory. Define both trained models’ URLs as constants and add functions which are used to detect the type of the device:
The next step is implementing the function to load model. This function is called in “app.vue” when the app starts. Install cordova.plugin.diagnostic with command npm i cordova.plugin.diagnostic, which is used to explicitly ask for microphone usage permission (line 7).
Line 20 uses ml5.js with model URL as a first parameter and threshold of the successful prediction as a second one. It is best to play around with the threshold to see which value will give the best results.
The last function to code is startPredition. As you can see, it has a callback as a parameter. That way we can access the result in other parts of the code.
If you start the application now, it accesses the microphone and starts predicting the piano chords. The next task is to deploy it on a mobile device. Check the last part of this tutorial, which will guide you step by step on how to use build a hybrid app in a few steps. Be aware that the prediction accuracy will vary depending on your phone’s/laptop’s microphone capabilities, and that the current implementation of ml5.js soundClassifier might not be able to access the microphone of all devices.
In case of any questions or suggestions for improvements, do not hesitate to contact me.
Good luck and have fun coding! :)
