Aksha: An Arduino based ML Pencil powered by TensorFlow Lite Micro
Zoom Call: A temporary portal for temples of Education
In India, Schools and Colleges are considered not less than temples, and in Hinglish, they are even associated as “Vidya kae Mandir” i.e. temples of Education.
However, due to the current ongoing pandemic, these temples closed their doors for their devotees and started looking for online solutions to continue providing education to their devotees.
Like many others, my sister's daughter's school also ended up using zoom call to conduct online classes. So, one fine morning I just took a break from my WFH schedule and went across the room where my sister’s daughter was attending her Hindi language class.
I observed that the teacher on the other side of the zoom call was musically reciting the Hindi alphabet which encouraged my sister’s daughter and other students to follow along but during the same day when my sister asked her daughter to write Hindi alphabets she was not ready to do it at the first go.
Then I realized that the teacher made the learning fun for my sister’s daughters but there are no ways to make that part of learning fun for toddlers when it comes down to writing.
Furthermore, this whole situation allowed me to cook a solution for this interesting problem and I ended up making Aksha The ML pencil for toddlers to draw Hindi alphabets in air powered by TensorFlow Lite Micro.
Purpose
After making you aware of Why I built Aksha an ML Pencil it’s time to demonstrate to the entire TensorFlow lite community and Embedded Machine Learning enthusiasts about How I built it.
Shifting Gears
Now from this point onwards, this article will shift from story mode to tutorial mode so all developers get ready, here comes your favorite part.
Hardware Requirements:
To follow along you’ll be needing the below-mentioned items:
- A laptop or desktop with a stable internet connection.
- Arduino Nano 33 BLE Sense Board
- Zebra Byte Case [*not necessarily required*]
- 1 x USB A to Micro USB Cable-120 CM [make data collection easy]
- Short USB to Micro-USB Power Line Cable-17 CM [for demo purpose I used it, not necessarily required, above mentioned cable can serve the purpose too.]
- USB-Hub
- Stick to hold the Arduino BLE board and the Micro USB cable together, you can look for something that is roughly 17-20 cm long.
- Kapton Tape
Demo SetUp A: Taping the micro USB cable on top of a stick with Kapton tape.
Demo SetUp B: the micro USB cable on top of a stick with Kapton tape with Arduino 33 BLE Sense Board.
Software Requirements
- Arduino IDE
- An up-to-date version of the Chrome web browser to use Web Bluetooth APIs.
Software libraries to Install
- ArduinoBLE [for communicating with the web page]
- Harvard_TinyMLx [for code base of Magic Wand]
- Arduino_LSM9DS1 [to read values of various sensors]
- Arduino Mbed OS Nano Boards (look in Board Manager for this package).
- Google Input tools [for data labeling]
Machine Learning Workflow
To construct Aksha the ML Pencil I outlined a workflow, which is more or less not different from any other Deep Learning workflow.
- Decide on a goal i.e. To build an ML pencil that will recognize 5 Hindi alphabets by analyzing their accelerometer and gyroscope data using Machine Learning.
- Collecting Data: through Web Bluetooth APIs.
- Model Training: on the collected Data using Google Colab.
- Deployment: Deployment of the trained model for 5 Hindi alphabets on Arduino Nano 33 BLE Board using Arduino IDE and TensorFlow Lite Micro
#Note: However, before starting with the above-endorsed workflow, to make the Arduino 33 BLE Sense Board capable of collecting data, I deployed Magic Wand Code on it, available through the Harvard_TinyMLx library in Arduino IDE.
Data Collection
After deploying the magic wand code on the Arduino Board, I navigated to https://tinymlx.org/magic_wand for data collection.
- Connected the Arduino board over Bluetooth, by clicking on the Bluetooth button.
- Then after pairing the board, I was ready to collect data through Chrome’s WebBLE API.
Recording Data
After setting up everything successfully, I started recording data for the Hindi Alphabets mentioned below, the glimpse of the same can be seen in the gif below.
Strategies I used for successful data collection:
- I didn’t switch the tab or refresh the page while recording the data in the browser.
- I recorded as many samples as I can in one go, normally I did 500–600 samples per alphabet.
{0: 'इ', 1: 'क', 2: 'ट', 3: 'न', 4: 'प'}
Labeling Data
After recording the data, I cleaned the noisy and not-appropriately recorded Hindi alphabets.
Then I switched to data labeling which would have been easy if I was labeling some English alphabets but Google Input tools came to my rescue and allowed me to label the recorded Hindi alphabets pretty easily.
Have a look at the below GIF, to visually understand the approach I followed to clean and label the recorded Hindi Alphabets data.
Downloading Data
For each alphabet, after completing the data recording, cleaning, and labeling step, I clicked on the Download button, and a JSON file got downloaded into my system. I generated 2 JSON files per alphabet to have a good amount of Hindi alphabets data for the ML Model training step.
Model Training
To train an ML model on the collected data, I cruised to the below link, made few changes added the collected data in the directory, and in the end got a TensorFlow Lite for Microcontrollers model for the Hindi Alphabets dataset, I collected in the previous step.
Model Deployment
Before deploying the model using the Arduino IDE on Arduino 33 BLE Sense Board, apart from the filenames of the Magic Wand code Module I made two specific changes in the codebase.
- In “hindi_alphabets.ino” I replaced the label count and the labels information with my Hindi Alphabets count and Labels Information
2. I replaced the entire content of the file “magic_wand_model_data.cpp” with the content of TensorFlow Lite for Microcontrollers model which is nothing but a “.cc” file generate after ML model training and named it as “hindi_alphabets_model_data.cpp”.
Then, with a lot of courage, I uploaded the code or in another way, I started compiling the Sketch to Arduino 33 BLE Sense Board.
After Compiling the code, to test the Hindi Alphabets Model, I opened the Serial monitor present in the Arduino IDE made few movements in the air using the Board and luckily the Serial monitor was able to correctly identify those movements as Hindi alphabets.
And that’s how Aksha: An Arduino based ML Pencil came into existence
The demo is included below, don’t miss it.
Resources:
- Github repo with tinyML examples can be found here.
- Pete Warden Tutorial: Building a Magic Wand.
- HarvardX’s Tiny ML Course.
- Introduction to Embedded Machine Learning
Github Repo:
The code, models, JSON file, videos, and most importantly, the entire Hindi Alphabets recorded data can be found here.
https://github.com/ElephantHunters/Hindi_Alphabets_Recognition.git
Use it and extend what I Built.
Gratitude Corner:
- Firstly a big shout out to Pete Warden who presented the tutorial: “Building a Magic Wand” at TinyML Summit 2021 which in turn motivated me to develop something simple, fun but most importantly a useful application of TinyML.
- Secondly, I would also like to thank Brian Plancher whose frugal advice allowed me to record data for Hindi alphabets in a better way.
