Face Detection on the edge in under 20 mins and $20

Evaluating the current state of low-power AI on the edge with Sipeed M1w AIOT modules

While scouting for a potential SoCs and SoMs for my next edge computing project, I had a list of things I was looking for -

1. Low-power consumption — Preferably >10 hrs on few AA cells
2. Low cost — Something that won’t be a novelty in an undergraduate hobby group
3. Small form factor — Should fit in a palm
4. Neural Network accelerator and easy to use with videos
5. Digital ports and buses you find on typical MCUs — i2c, spi, uart…
6. Networking capability via WiFi

And I wanted all of these with an easy to use developer friendly kit! Too tall an ask?

To my delight (and honestly, utter surprise), the Kendryte 210 SoC seemed to fulfil all my computing and power requirements. It promises a computing performance comparable to Jetson Nano at a fraction of price, power requirement and form factor! Moreover, the Sipeed Maix module (funded through Indiegogo) packages the K210 SoC in a feature rich unit and provides a suite of developer kits and software built over these module. But, should I use this magnificent looking kit or retain the option to script in Python like I would have on a Raspberry Pi or Jetson?

Sipeed supports development in micropython, which really flattens the learning curve for Python software developers and AI engineers looking to venture into AI on the edge. There are lots of example projects online showcasing real-time classification and detection running on Maix developer kits (it promises up to 60fps in VGA!). But, as Abraham Lincoln had rightly said, we must not trust everything we see on the internet. So, I had to try this out for myself. Try I did, and I was blown away.

The only complaint I had from the whole experience was with the documentation — scanty and in broken English. It seems the Maix documentation has been a common concern among makers around the world. However, this seems to be improving too. It took me around 5 hours from unboxing to understanding the ecosystem and getting the face detection example running (not too bad). But, I believe it can be better if a few things were introduced in advance, and hence this article. This guide is based on the official MaixPy documentation, with a few details here and there to save you time and setup your development environment while testing the example.

Getting the hardware

Here’s that hardware and platform you will need to begin:

1. A Maix development board and peripherals — I suggest Maix M1w Dock suite, available for $19.90 at seeedstudio on the day this was written. The ‘w’ in M1w means it has WiFi capability. Additionally, the kit comes with an LCD and a camera, so you can kick start in no time.
2. A USB Type C cable or a USB Type B data cable+Type B to C adaptor. The Maix board has a type C female jack, so do the math. Make sure you use a data cable and not a charging cable. Some cables are meant only to be used as charging cables and the board will not connect to your computer if you use those.
3. A computer running Ubuntu 18.04 (16+ should be fine). You do not need to install drivers if you use the Dock suite but you might need to add your user account to the dialout group by using the command sudo usermod -a -G dialout $(user) (replace $(user) with your user name).

If you have all the resources in the list above, it should take less than 20 mins to implement the example and get comfortable with the development ecosystem. So let’s get started!:

Collecting Resources

Some of the links mentioned below maybe a minute or so to open, so be patient.

1. Download the Kflash GUI to flash firmware on to your board. I used the 7zip file in version 1.55 available here — kflash_gui_v1.5.5_2.7z. Extract the 7z file and you will get a folder with the kflash_gui executable. Provide executing permission to the executable file using chmod +x kflash_gui.
2. Download the MaixPy IDE as we intend to learn how to write our own programs in micropython. I used the installer for Linux for version 0.25 available here (download the file with the extension .run). Execute the command chmod +x maixpy-ide-linux-x86_64-0.2.5.run in your download folder and then run ./maixpy-ide-linux-x86_64-0.2.5.run.
3. Download the example Maixpy scripts.
4. Download the resources at the face detection example blog. We will be using the face detection model from this resource (download this zip).
5. Download a firmware for the board to update the default firmware it shipped with. I have used firmware version 0.5 with lvgl support available at — maixpy_v0.5.0_62_gd6620f1_with_lvgl.bin.

Code and Flash!

Now we are all set to go through the steps to flash our first program!

1. Power up the board — Connect the camera and lcd to the board at respective slots. There is small marking named “LCD” on the board to avoid any confusion. Connect the USB cable to the board and it immediately powers up. With the usual firmware shipped with the board, it should either display the camera feed in the LCD, or display a message similar to this:

Welcome to MaixPy

Additionally, if you have no other USB devices connected to you system executing the command ls /dev should list /dev/ttyUSB0 in the list of devices.

2. Flash the firmware — The firmware we downloaded contains basic functions for the board and the chip, along with support for micropython, MaixPy IDE, machine vision and display. Run the kflash-gui tool, and select the firmware .bin file, serial port, board setting as follows:

Typical Settings

Hit the Download button and wait for the firmware transfer to be completed. With this step, your board is loaded with the most commonly used libraries, and is able to run REPL over serial terminal and connect to the MaixPy IDE.

3. Test out the micropython console: This step will ensure we have loaded the firmware correctly and give a brief idea about what we can do with micropython. We can use any serial terminal (like minicom), but we will use the one provided with the IDE. To connect, open up the MaixPy IDE and from the top menu, select ‘Tools > Open Terminal > Serial Port — ttyUSB — 115,200 BPS’. This will open a the micropython terminal similar to this:

The MaixPY Terminal

If you are not presented with the console by default (you are not able to see >>> at the last line in the terminal window), it means there is a script already running. In that case, just press Ctrl+C to stop that script and you will get the prompt. Type print(‘hello world’) in the console and press enter. It should print the string ‘hello world’, just like in a python shell!!!! Now, you have a system that runs micropython and you can write scripts on it, upload custom data files and use it just like you code in Python (similar, not same, to be precise).

4. Flash the neural network model: Unzip the file we had downloaded in step 5 of ‘Collecting Resources’ section. You will find 2 files in the extracted folder — ‘face_3M.kfpkg’ and ‘maixpy_yolo2.bin’. Since, we have already uploaded a more recent fimware in Step 2 of this section, we only need to download the neural net model to the board. To do so, select the .kfpkg file in the kflash_gui tool and press the ‘Download’ button keeping other settings same.

5. Code and Run!: Go the MaixPy IDE and select ‘File > Open File’. Navigate to the folder where you have the Maixy Scripts saved (step 3 of Collecting Resources section). Inside the folder named ‘machine vision’, select and open the file ‘demo_find_face.py’. This is the micropython script, which will use the resources from the firmware and the neural network model we uploaded to run a face detection model. There are two ways to run this code and we will try both:

• On the bottom left corner of IDE, use the connect button and select the correct port to connect to the board. Then click the run button below it.

You can see the display on the host computer while it is connected to the IDE

And that’s it! Within 25 lines of code, we are able to run a face detection network, and display the results in real-time both on the LCD and on the laptop screen! If you have trouble connecting, make sure you have closed the terminal to ttyUSB0 that you used in step 3(you can only have one active connection at a time).

• Now stop the code and disconnect using the buttons on the bottom left corner of the IDE. From the top menu, select ‘Tools > Save open script to board(boot.py)’. Every time you restart the board, boot.py is the first program to get executed. So, with every boot, your face detection script starts running by default and displaying the result on the LCD. Since, we are not connected to the IDE anymore, this is faster than the way we executed the same code previously (I will leave the FPS tests to you).

Few Important Notes

I hope you are excited about immense range of flexibility this power packed platform gives to implement your next project. You just need to write your own scripts and create your own neural network models and let the Maix module handle the rest. To get you there quicker, I will answer some basic question I had in my mind when I first tried this platform:

1. What’s up with the ‘boot.py’ file?: Maix follows a standard way to implement start up behaviour (similar to most micropython systems):

• If a valid microSD card (in FAT format) is found and it contains a ‘boot.py’ file, this is the first file to be executed.
• If there is no SD card, the system looks for a ‘boot.py’ on the internal flash (which is where the IDE uploads it) and executes it.
• Once this script is executed, the system looks for a ‘main.py’ file and starts executing it.

So, assuming you are not using an SD card, a good practice would be to write your board configuration in ‘boot.py’, your main loop in ‘main.py’ and all other modules in different files and importing them as needed. You can send the files to the board directly using the ‘Tools’ menu in the IDE, and delete using os.remove() in the serial terminal.

2. And what is the .kfpkg file?: We uploaded a .kfpkg file in Step 4 of ‘Code and Flash!’ section. You might also have noticed that we used the kflash_gui tools twice. Wouldn’t that overwrite the firmware we flashed first? Nope — and this is where .kfpkg files come in. It is nothing but an archive (a zip file) with its extension renamed to .kfpkg instead of .zip. Try unzipping the ‘face_3M.kfpkg’ we downloaded as part of face detection model. It will contain 2 files — a .kmodel and a flash-list.json file. The .kmodel file is the neural net, while the .json file tells our kflash tool where to upload it.

The json file says upload the facedetect.kmodel file at the given address in flash memory, and do not flash or verify the sha.

Now look the script we ran on our board:

Look at it!

On line 12, you see the KPU neural net accelerator reads the neural net from the same address where the json file asked the kflash_gui to upload it!

Here’s an exercise to bundle Step 2 and 4 of ‘Code and Flash’ section :

• Put the .kmodel, .bin firmware and .json file in same folder.
• Create an entry in the .json file for your .bin file, and mark it to be flashed at the beginning of the flash like in the image below:

We have added another file to file-list.json. Good practice for .bin files to keep sha prefix true.

• Zip the folder and rename the extension to .kfpkg. Now you can upload a single .kfpkg and it will deploy both the firmware and the neural net at the correct locations in the flash in one go!

3. How do I make my own models?: The K210 SoC documentation details the capabilities of the KPU neural net engine. You can train your deep learning models in tensorflow and obtain a tflite (tensorflow lite) model. Then you need to use nncase to convert the tflite model to .kmodel, which goes on your Maix boards. You can follow this blog to learn more about how to do that.

The objective of this article was to quickly get you started with AI on the edge even if you have never worked with embedded systems before. I hope you enjoyed following this article and are looking forward to sharing your projects based on Maix boards. Happy Coding! :)

Please reach out if you have any doubts or if you would like me to elaborate on or modify any section of the article.

Acknowledgement: The contents of this article were made possible with the resources acquired as a part of exploratory project for (Milky Way AI), for whom I currently serve as the CTO.