Doing Machine Vision on Google Coral with OpenCV

Published in

KrakenSystems

5 min readJan 27, 2020

Doing Machine Vision on Google Coral with OpenCV

While developing computer vision applications for embedded devices, I realized that execution time is going to be the number one priority. However, after failing to install all required libraries for such a task, I understood that an easy setup process is going to be of great value as well. What I am specifically talking about is installing OpenCV for Python applications on Google Coral Dev board. Coral was meant to run inference at incredible speeds. But of what use is Coral to me if I can’t even process video frames with OpenCV?

While everything worked flawlessly on a desktop computer, I went to great lengths when installing OpenCV 4.1.1 on Coral. If you are in the same situation, you might have just found a solution! While there is a guide which I followed thoroughly, it didn’t quite work for OpenCV 4.1.1.

Therefore, this post explains how to install OpenCV 4.1.1 on Google Coral Dev board and it is an extension to this and this guide, but with changes and explanations of what and why I changed it.

Requirements

Here is a list of requirements you should do before following this guide:

Flash a system image on the board.
Connect to your device using either ssh or serial console.
Install OpenCV dependencies. Here is how to install everything that is needed for a successful OpenCV build:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install build-essential cmake unzip pkg-config
$ sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
$ sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
$ sudo apt-get install libxvidcore-dev libx264-dev
$ sudo apt-get install libgtk-3-dev
$ sudo apt-get install libatlas-base-dev gfortran
$ sudo apt-get install python3-dev

Install git:

sudo apt-get install git

Quick solution

Download the prebuilt libraries from this repository.

git clone https://github.com/pjalusic/opencv4.1.1-for-google-coral.git

Copy the cv2.so file to /usr/local/lib/python3.7/dist-packages/.

cp opencv4.1.1-for-google-coral/cv2.so /usr/local/lib/python3.7/dist-packages/cv2.so

Copy other .so files to /usr/local/lib/.

sudo cp -r opencv4.1.1-for-google-coral/libraries/. /usr/local/lib

Check if it works.

python3
>>> import cv2
>>> cv2.__version__
'4.1.1'

If it works, you are done. You can now link cv.so file to your virtual environment if you’d like. However, if you prefer building OpenCV yourself or it didn’t work using the quick solution, here is a guide.

Build your own libraries

Create a swap file for the build.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Google Coral has only 1GB of RAM which is less than required for the build job. Therefore, we create a swapfile. I am using a 2GB swap file to eliminate any possibility of errors while building, which occurred when using only 1GB swap file.

Mount USB formatted to ext file system (for formatting try this for mac, or this for linux).

sudo /sbin/fdisk -l # at the end is your device 
sudo mount /dev/sda1 /mnt

Instead of USB you can use an SD card. Whichever works, but it should have at least 4GB of memory.

Download OpenCV 4.1.1.

$ cd /mnt
$ wget -O opencv.zip https://github.com/opencv/opencv/archive/4.1.1.zip
$ wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.1.1.zip
$ unzip opencv.zip
$ unzip opencv_contrib.zip
$ mv opencv-4.1.1 opencv
$ mv opencv_contrib-4.1.1 opencv_contrib

Make changes in opencv/cmake/OpenCVFindLibsPerf.cmake file. Before compiling OpenCV, you need to do something extra to avoid errors. This might be only necessary for version 4.1.1, but it may help with others as well. I created a public gist containing a git diff patch file. Clone the repository into your cmake folder and apply changes.

$ cd opencv/cmake
$ git clone https://gist.github.com/pjalusic/7feb9cd722a437c0f6eea34622ca44c8.git
$ cp 7feb9cd722a437c0f6eea34622ca44c8/OpenCVFindLibsPerf.diff .
$ git apply OpenCVFindLibsPerf.diff

If there are errors when applying the patch, double check you downloaded OpenCV 4.1.1 or do this by hand. There are exactly 5 lines commented. If it was successful, part of OpenCVFindLibsPerf.cmake file from line 43 should look like this:

if(WITH_EIGEN AND NOT HAVE_EIGEN)
 find_package(Eigen3 3.0.0)
 if(Eigen3_FOUND)
   #if(TARGET Eigen3::Eigen)
     # Use Eigen3 imported target if possible
     #list(APPEND OPENCV_LINKER_LIBS Eigen3::Eigen)
     #set(HAVE_EIGEN 1)
   else()
     if(DEFINED EIGEN3_INCLUDE_DIRS)
       set(EIGEN_INCLUDE_PATH ${EIGEN3_INCLUDE_DIRS})
       set(HAVE_EIGEN 1)
     elseif(DEFINED EIGEN3_INCLUDE_DIR)
       set(EIGEN_INCLUDE_PATH ${EIGEN3_INCLUDE_DIR})
       set(HAVE_EIGEN 1)
     endif()
   #endif()
 ...

As you can see there are a few changes. The first one is using Eigen3 3.0.0 instead of Eigen QUIET. Secondly, we comment out one if check which gave errors. What all of this does is fixes the compatibility problem with Eigen library because CMake can’t see it. Mind the last comment on endif().

Create a python virtual environment.

cd ~
wget https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py
sudo pip install virtualenv virtualenvwrapper
sudo rm -rf ~/get-pip.py ~/.cache/pip

Add the following lines to ~/.bashrc file (ie. nano ~/.bashrc) for virtualenv and virtualenvwrapper.

export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

Source that file.

source ~/.bashrc

Create a virtual environment, activate it and install numpy (for some reason I was getting errors until I connected to the device through ssh).

mkvirtualenv cv -p python3
workon cv
pip install numpy

Compile OpenCV 4.1.1.

$ cd /mnt/opencv
$ mkdir build && cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
  -D CMAKE_INSTALL_PREFIX=/usr/local \
  -D INSTALL_PYTHON_EXAMPLES=ON \
  -D INSTALL_C_EXAMPLES=OFF \
  -D OPENCV_ENABLE_NONFREE=ON \
  -D OPENCV_EXTRA_MODULES_PATH=/mnt/opencv_contrib/modules \
  -D PYTHON_EXECUTABLE=~/.virtualenvs/cv/bin/python \
  -D ENABLE_FAST_MATH=1 \
  -D ENABLE_NEON=ON -D WITH_LIBV4L=ON \
  -D WITH_V4L=ON \
  -D BUILD_EXAMPLES=ON ..

Start building.

make -j$(nproc)

This will utilize all cores for the build process. It, however, still takes a couple of hours.

Install OpenCV.

sudo make install
sudo ldconfig

Link to the virtual environment.

$ cd /usr/local/lib/python3.7/dist-packages/cv2/python-3.7
$ sudo mv cv2.cpython-37m-aarch64-linux-gnu.so cv2.so
$ cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
$ ln -s /usr/local/lib/python3.7/dist-packages/cv2/python-3.7/cv2.so cv2.so

Test it.

workon cv
python
>>> import cv2
>>> cv2.__version__
'4.1.1'

Well, this was a bit painful and a long process that took too much time. Obviously, the next step is to speed up the build process. How? Well, an idea is to move the build process to an ARM device in the cloud. But that’s a story for another blog post.

Additional information, notes and problems

Build OpenCV in background

If you would like to do any process in the background, you should check tmux or similar tools. Here is how to do it using tmux:

sudo apt-get install tmux
tmux → creates a new session. Run workon cv once again. Run make -j$(nproc) command while in the session
Control+b, d → leave session
tmux ls → show sessions
tmux attach -t 0 → return to that session

Permission errors while building OpenCV

If you encounter permission errors while compiling, building or installing OpenCV, you could always go with the root user option. So before building or opening session with tmux do sudo su. That will also mean that any sessions you create will be bound to root user.

Problem with reflashing

I stumbled on a serious issue when trying to reflash the device to test my tutorial once again. Following this official guide on reflashing, my Mac was stuck at mdt devices step. Nothing was showing up. Apparently, it is an ongoing issue. My solution was just switching to Windows computer. To follow the guide and successfully reflash the device all I needed to do was:

Install python3.
Install mdt with pip3.

pip3 install --user mendel-development-tool

Get fastboot tool from SDK Platform Tools and manually add fastboot.exe to path

Credits

Originally published at https://krakensystems.co on January 27, 2020.

Doing Machine Vision on Google Coral with OpenCV

Doing Machine Vision on Google Coral with OpenCV

Requirements

Quick solution

Build your own libraries

Additional information, notes and problems

Build OpenCV in background

Permission errors while building OpenCV

Problem with reflashing

Credits

Written by Petar Jalušić