Visual Wake Words on Arduino Nano
Person Detection model deployed to tinyML devices effortlessly
Are you interested in making $20 device smart by running deep learning model with 82KB size to detect a person? This device can be incredibly useful to detect a person in front of a door, shop, buildings, offices, factories, or to avoid person in restricted areas such as forests, cell phone towers etc.
I created a dataset by clicking 500 photos. Fed the dataset to a TF model and achieved 87% accuracy in cAInvas notebook. The trained model was compiled with deepSea compiler. The compiled library from deepSea was integrated in an application. The resulting binary was flashed to Arduino Nano board using Arduino IDE. Watch this process in action.
Technical Details
Running deep learning models in such tiny devices is a challenge for two reasons. First, tinyML devices such as Arduino Nano 33 are fairly resource constrained in terms of CPU cycles, RAM, storage space etc. Second, the runtimes for popular ML frameworks aren’t available on these tinyML devices. TF Lite Micro has an implementation but that lacks support for many operators such as RNNs, LSTMs etc.
AITS’ deepSea compiler takes in a deep learning model in Keras, TensorFlow protobuf or ONNX format. It creates a package which includes the C++ library, required header files and a readme that shows how to use the library. The deepSea compiler is optimized for tinyML devices with ARM, Xtensa and many other HW architectures.
To create the Arduino application, I started with ArduCAM example ArduCAM_Mini_2MP_Plus_functions. I removed non relevant code for my use case. Link to my modified .ino file is here.
Once you have followed the instructions provided below in Software Setup section and created the deepSea library from the cAInvas platform, please follow the instructions on github on how to put the library such that Arduino IDE finds the visual_wake_word.a library to link to your application.
Hardware used:
- Arduino Nano 33 BLE Sense with Headers — prefer one w/ Headers so you won’t have to do the soldering etc.
- ArduCAM Mini 2MP Plus — I preferred this because there’s only 8 wires to deal w/ vs. some other cameras where you might need to deal with more than 15
- You will need some female to female jumper wires (shorter the better)
- A stand of some sort to hold the camera straight would be ideal
- USB-A to micro-USB cable
Software used:
- cAInvas — End-to-end tinyML development platform from AITS
- deepSea — AITS’ ML compiler to convert models to C++ std11 library
- Arduino IDE — IDE to program Arduino and other tinyML devices
- ArduCAM library — Library to take still images from ArduCAM camera
- JPEGDecoder — A JPEG Decoder Library
Hardware Setup:
- The diagram below shows how the ArduCAM needs to be connected with Arduino. It’s preferred to have ArduCAM connected directly with female-to-female wires vs. through breadboard.
Software Setup:
- Download and install Arduino IDE
- Install supporting libraries for Arduino Nano 33 BLE Sense — link
- Install ArduCAM libraries in Arduino IDE — link
- For ArduCAM 2MP Plus camera used in the demo, under ~/arduino/libraries/ArduCAM/memorysaver.h file, select hardware platform by uncommenting ‘#define OV2640_MINI_2MP_PLUS’. All other hardware platform defines must be commented. I did not use “UTFT4ArduCAM_SPI” library as I did not connect a TFT display.
- Install JPEGDecoder library in Arduino IDE by traversing the menu as follows: Sketch → Include Library → Manage Libraries. Search of JPEGDecoder by “Bodmer, Makoto Kurauchi, Rich Geldreich” and install v1.8.0
References:
- Free signup on cAInvas
- ASL Recognition with TinyML Devices using cAInvas
- PyTorch sample notebook for ASL detection in PyTorch
- Speed-Up Pre-Trained Deep Learning Models on Micro-Controllers
- Deep Learning with deepC DNN Compiler — part 1
- Deep Learning with deepC DNN Compiler — part 2
- Arduino Nano 33 support on Arduino IDE
- ArduCAM Arduino GitHub repository
- JPEGDecoder Library
