Sign language recognition on Android with Intel OpenVINO

4 min readDec 2, 2020

This article explains how to create a sign language recognition application on Android x86 OS (64-bit) with Intel OpenVINO. We used the core component of OpenVINO — Inference Engine, which manages the loading and compiling of the optimized neural network model, runs inference operations on input data, and outputs the results.

There are multiple Android OS distributions which may run on Intel Architecture. Some of them are:

The application reads a video from the camera, collects every 16 frames in a sequence, uses a neural network to recognize a sign word and displays a text with the recognized word on image.

Build OpenVINO Java bindings for Android:

These steps were done on Ubuntu 18.04, but in the general case, they can be done on other operating systems, without fundamental differences.

Download and unpack Android NDK to ~/Downloads folder.
Install OpenJDK 8:

sudo apt-get install -y openjdk-8-jdk

Export OpenJDK 8 path:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Clone OpenVINO repositories to your computer. Let’s assume that ~/Downloads is used as a working folder:

cd ~/Downloads
git clone https://github.com/openvinotoolkit/openvino.git
git clone https://github.com/openvinotoolkit/openvino_contrib.git

Update openvino submodules and create build directory:

cd openvino
git submodule update --init --recursive
mkdir build && cd build

Build OpenVINO for Android:

cmake \
  -DANDROID_ABI=x86_64 \
  -DANDROID_PLATFORM=21 \
  -DANDROID_STL=c++_shared \
  -DENABLE_VPU=OFF \
  -DENABLE_GNA=OFF \
  -DENABLE_CLDNN=OFF \
  -DENABLE_OPENCV=OFF \
  -DENABLE_SAMPLES=OFF \
  -DIE_EXTRA_MODULES=~/Downloads/openvino_contrib/modules \
  -DCMAKE_TOOLCHAIN_FILE=~/Downloads/android-ndk-r20/build/cmake/android.toolchain.cmake ..make --jobs=$(nproc --all)

To reduces size of the binaries do:

~/Downloads/android-ndk-r20/toolchains/llvm/prebuilt/linux-x86_64/x86_64-linux-android/bin/strip ../bin/intel64/Release/lib/*.so

Create Android Studio project

Download Android Studio on your PC.
Start a new project, choose “Empty Activity”

Modify the following project files:

/app/src/main/AndroidManifest.xml

/app/src/main/res/layout/activity_main.xml

/app/src/main/java/com/example/aslrecognitionapp/MainActivity.java

Add Inference Engine dependency to project

You need the following files from OpenVINO:

cd ~/Downloads && mkdir openvino_installcp openvino/bin/intel64/Release/lib/plugins.xml openvino_install
cp openvino/bin/intel64/Release/lib/inference_engine_java_api.jar openvino_installcp openvino/bin/intel64/Release/lib/*.so openvino_installcp openvino/inference-engine/temp/tbb/lib/libtbb.so openvino_install
cp openvino/inference-engine/temp/tbb/lib/libtbbmalloc.so openvino_install

Also you will need C++ runtime library:

cp android-ndk-r20/sources/cxx-stl/llvm-libc++/libs/x86_64/libc++_shared.so openvino_install

Create jniLibs/x86_64 directory in ~/AndroidStudioProjects/AslRecognitionApp/app/src/main folder:

mkdir -p app/src/main/jniLibs/x86_64

Copy native libraries to the project:

cp openvino_install/*.so app/src/main/jniLibs/x86_64

Copy inference_engine_java_api.jar file to the ~/AndroidStudioProjects/AslRecognitionApp/app/libs folder:

cp openvino_install/inference_engine_java_api.jar app/libs

Add inference_engine_java_api.jar dependency in Gradle Scripts -> build.gradle (Module: AslRecognitionApp.app) file:

implementation files('libs\\inference_engine_java_api.jar')

Add OpenCV dependency to project

Download OpenCV SDK for Android.
Import OpenCV module: File -> New -> ImportModule and specify a path to unpacked SDK: ~/Downloads/opencv-4.5.0-android-sdk/OpenCV-android-sdk/sdk

Add module dependency: File -> Project Structure

Replace minSdkVersion 16 to minSdkVersion 21 in Gradle Scripts -> build.gradle (Module: AslRecognitionApp.app) :

Download deep learning network

Download model files from Open Model Zoo :

git clone --depth 1 https://github.com/openvinotoolkit/open_model_zoocd open_model_zoo/tools/downloaderpython3 -m pip install -r requirements.in
python3 downloader.py --name asl-recognition-0004

Model will be downloaded to intel/asl-recognition-0004/FP32 folder.

Add files on Android device

Use Android Debug Bridge (adb) to transfer data files on Android:

adb push ~/Downloads/openvino/plugins.xml /data
adb push asl-recognition-0004.xml asl-recognition-0004.bin /data

Try to run the application

The first time application will ask for camera permissions. If you’ll see a blank screen after that — try to run application again.