Developing a Face detector Android app using Unity3D and OpenCV-CPP plugin

Ever wanted to develop any applications for edge devices? Ever wanted to leverage efficient libraries to build real-time applications. This blog will help you understand and leverage the efficiency of cpp to build an efficient android app using unity3d. The method mentioned here is a generalized one and can be used to create similar ones.

For quite some time, I was trying to develop an android based application for general face detection. I had some old C++codes written using OpenCV. I was exploring different ways of creating it. One obvious step is to port complete c++ code to java and use android studio to build a native android app. Since most developers working in AR/VR or game industry use Unity3d it will be really convenient for them to build advanced UI applications. But the real challenge is working with matrix operations and other heavy computing stuff, which you will encounter, building computer vision or any DL/ML-based applications. writing heavy computing operations could be easy and efficient using cpp. Hence, In this article, I mainly focus on developing an android application using Unity3D and Native CPP.

Compiling OpenCV for Android

The first step towards this is to compile OpenCV from src for android.

1. Download OpenCV from Github.
2. Select “MinGW Makefiles” as the generator
3. Select “Specify toolchain file for cross-compiling”
4. Then provide the toolchain path like this

5. Disable BUILD_ANDROID_EXAMPLES, BUILD_ANDROID_PROJECTS, BUILD_TESTS, BUILD_PREF_TESTS, BUILD_FAT_JAVA_LIB and enable BUILD_SHARED_LIBS

6. Then use “mingw32-make” command to build the shared libraries

7. Use “make install” to install the libraries in a specific location

Note: Here I was using visual studios ndk library toolchain instead of the one that is shipped with OpenCV to avoid the issue of clang and gcc compatibility

Creating Face Detection Library for Unity3d

Now that we have OpenCV shared libraries build for android, its time to used to create a shared library for unity3d.

1. Setup Visual Studio (I use visual studio 2019) with android development environment
2. Create “Dynamic shared Library Android”
3. Configure with built OpenCV library paths and include paths
4. Set the Configuration type to “Dynamic Libray (.so)”

5. Add include path in “VC++ Directories >> Include Directories”

6. In “Linker >> Addition Library Directories” add the library path

7. Add libraries to “Linker >> Input >>Library Dependencies”

Once, configuring is done we are good to go to compile our existing code to build a shared library.

sharedocv.h file

sharedocv.cpp

“sharedocv.h” file implements the function definitions and “sharedocv.cpp” file implements the functions and export functions for shared library.

Creating an Android app using unity3d

1. Configure Unity3d for Android. you can check this. There are lot of articles out on the internet to configure unity3d for Android in case you face difficulty.
2. Create a Canvas and Add RawImage and Buttons, Text like below

3. Create Script to capture video and detect face

4. Attach Script to “EventSystem ”and link objects like below

5. Now Build the application with “IL2CPP” instead of “Mono”. To change go to “Player Settings >> Configuration >> Scripting Backend”.

Check my Github for the Complete project.

If you like to know more about developing efficient, real-time applications for edge devices please connect with me on LinkedIn.

References:

1. https://amin-ahmadi.com/2019/02/03/how-to-build-opencv-4-x-for-native-android-development/
2. https://github.com/spmallick/learnopencv/blob/master/FaceDetectionComparison/