As an industial designer I make my own low cost machine vision Hardware
TL;DR : The story of how I involved in the maker movement, Learn to build projects and finally group up a team Named HICAT, then make a low cost machine vision hardware platform called Livera, it on kickstarter now.
My name is Joseph Wang, background in industrial design. Thanks to the Maker movement, which let me, as a designer, could have access to Arduino and various technology.
As a industrial designer, I always looking for pain points in life any try to solve it via design a new form of the product or user experience, in order to make life easier. By gaining the skills of electronic, I feel like just turn into 170mph highway.
When I was doing a exchange program in RMIT, I started to build my first project in a course called “make things interactive” by lecturer Chuan, during this fundamental, detailed course, I’ve acquire the basic notion of Arduino project development. Finally the project end-up to be a ferry wheel like shoebox demo，which can store 6 pair of shoes. The wheel can turn to specific angle and expose the shoes you want, by knocking a pre-settled string of beat on the shoebox.
The shoebox project let me past the course, but the process was quite painful. After burned down several Arduino UNOs, I’ve become much familiar with the electronic development workflow, like how the MCU works, how to write program loops to track sensor data, and react to that data by control the motors to move specific angle. After all, it gave me more confident to go further.
In the second semester, I’ve join another course call “Electronic and Interactive Protopyting” by lecturer Scott, with previous experience, I was more vigourous. We had been introduced Electric imp, its a wifi enabled dev board with cloud intergrate, which means it can be controlled and sent data remotely through internet.
I decided to do crazy project, which is a dog tracker that can record live image and activity level of my dog, and stream it to me remotely, also interact with me through twitter. The project was named dogtime.
In hardware building, I choose an TTL serial Camera, accelerometer and Temperature sensor, order it from Adafruit, solder it onto a perf board to let all the pins been connected to the right port, then I started doing research on software building.
It was pain to develop on a brand new platfrom. it has its own workflow, even the programming language is brand new. After a painful moment, both of accel and temp sensor were working well, but it just couldn’t grabbed photos from the camera, it should be some kind of compatibility issue, I had to looked through all the documents and forum discussion, some said that I should use SPI port and other said Serial port, and I have tried both it just cloudn’t work. Therefore, my project ended up without camera module.
When I backed to china, I’ve graduated and got a job as a application engineer in Seeedstudio.
I’ve keep gainning skills by play around with different opensource hardware, built applications on top of it, like Deskpomo which is a Arduino enabled desktop pomodoro timer which aims to help people improve their wokring productivity, with the pomodoro method.
At that time, Esp8266 was the freshest and hottest development platform in the world, I just got one and immediately hook a cam module on it, but as the same result, failed.
After that, I started doing a serious research, I’ve look through almost all the existed camera module across TI, NXP, STM32, and accidentally, my roommate bought a Xiaomi security webcam，it is a wifi webcam which could stream high resolution video wirelessly and the it had a SD card port for local storage. I got his approval to disassemble it, when I saw the PCB, I thought it had all the things I need.
The core of the PCB was Hi3518e, it a SOC that runs embedded linux. I had read through the SDK and some other documents or applicant on internet, I find it was perfect and decided to start the project.
I called out some makers, talked about the idea to them, and finally I got a hardware and a software genius’s attension, after several detailed meeting, we’ve grouped up, named us the Team Hicat, and started the project Livera, which means Live + Camera.
After the project Livera started seriously, I’ve wrote a draft about the request list of basic architecture of Livera. First of all, I thought the most meaningful thing for Livera was to bring machine vision features to Arduino community, so there gonna be are duo core system, a vision core for image processing and recording, an Arduino core for development within Arduino IDE. Finally, the Hi3518e for vision core has been connected to Atmel 32u4 , via Serial port, in this way, we can provide simple APIs for Arduino developers, enables them to make any visual based projects, by upload codes to 32u4 through usb port, Also, the vision core is unique, you could access into it via ssh or serial debug port on board, that makes it whole system really flexible.
For the camera module, I wanted it to be unique and extendible, so that you could choose to plug directly on board or expand it via cable, depends on your projects.
Apart from the main components, we’ve added other parts like St7601 wifi module, Mpu6050 6-axie accelerometer, SD card slot, Ceramic antenna, which brings various functions.
A motor driver with 2 mos tube based H-bridge which able to power up 4 channels of DC motors and 4 channels of Servos, as the first shield of Livera.
The last issue for hardware bit was the board size, I wanted the PCB size could be as small as possible, after balancing with Heat dissipation and antenna signal，the final size is nearly a half cut of Arduino Uno at 60x42mm with 1.6mm 4 layer pcb.
As for the software bit, it can be divided into two part, one for vision core’s firmware, another for mobile app. The mission for vision core is video processing and capturing, also this embedded linux system can handle network streaming, so we’ve built up an RTSP server to handle video and file streaming, the server that could receive command from both Arduino core, web interface and linux cli, which means you could have multiple ways to capture image, that really make things easier.
For the Arduino bit, we have to wrote a library which contains APIs for visual based mechanism like(1.0 version haven’t finish yet):
The Robot kit
6 month since project started, after 2 batch of pcb test model, Livera become stable. I’ve played around with it on hand, watching this amazing little guy for hours, thinking about what I could do with it, tons of ideas just come out of my brain, dog feeders, pet trakcers, security door bell, at the end, I figure out what was the best application for Livera, it should be a Robot.
After 1 day of research, I find people use openCV library to figure human traffi on Hi3518e. Imagine that a robot could realize the context around, and perform certain task, and streaming the liver video to you as a extension of your eyes, how cool it gonna be.
I instantly open Coredraw started design a laser cutted acrylic frames, gather Livera, Motor driver, some motors and other essensial parts to build the robot.
The robot has 2 wheels and 1 Vientiane ball，the camera module has been attached to a servo arm which enables viewing up and down, it has a small size but works really well.
With the openCV library, the robot could do object and color tracking, even face recognition. It has the potential to be a great universal robot platform.
Another 4 months effort to finish all the preperation for KS campaign, the post shows what I tought to be the advantages of Livera:
- 720hd video and photo capturing&processing
- OpenCV enabled Color-Sensitive and Object-Tracking.
- Wifi enabled wireless control (this can work with the iot!)
- Image capturing and data recording onto the onboard sd card
- Mobile observation and manipulation from our custom apps( web-based and native)
- Programmable and open source.
It is always exciting to think about the future, as for the buzz of AI recently, undeniably, it should be the future.
When I saw the article of How a Japanese cucumber farmer is using deep learning to sort cucumbers. I think Livera should be best machine vision front-end for image capturing and openCV pre-processing.
With the natural language virtual assistant and could deep learning platfrom, we could just teach the robot how to perform task by speaking, or let him learn by watch, that will be really awesome, like people could finally owns their real world R2D2 and BB8.
- more advance image capturing and processing capability
- natural language virtual assitant
- deep learning could