Body Skeleton Tracking with Huawei AR Engine

Body Tracking

Introduction

The use of Augmented Reality (AR) is increasing every day in many areas from shopping to games, from design to education and more. If we ask what augmented reality is, we will get an answer from wikipedia exactly as follows.

What is the AR?

Augmented reality(AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.

However, today we will talk about the advantages of AR Engine, the augmented reality SDK offered by HUAWEI, rather than this classic definition. We will talk about the differences with the ML Kit, which looks similar but different, offered by HUAWEI, and we will develop a demo application using the HUAWEI AR Engine. And in this way, we will learn how to easily integrate augmented reality to our application by using HUAWEI AR Engine.

What is the HUAWEIAR Engine?

HUAWEI AR Engine is a platform for building augmented reality (AR) apps on Android smartphones. It is based on the HiSilicon chipset, and integrates AR core algorithms to provide basic AR capabilities such as motion tracking, environment tracking, body tracking, and face tracking, allowing our app to bridge virtual world with the real world, for a brand new visually interactive user experience. AR Engine accurately understands and provides virtual and physical convergence capabilities for our applications.

AR Engine Advantages

Normally, integrating augmented reality features to our application is a very complicated and laborious task. However, companies have offered SDKs to make this complex work easy. Apple AR Kit, Google AR Core and HUAWEI AR Engine, which is our main topic to develop applications today, are examples of these SDKs. However, there are differences between these SDKs such as performance, capability, and supported devices.

For example, while Google AR Core does not support face tracking and human body tracking, it does support AR Engine and AR Kit. Also, while AR Engine and AR Kit support hand gestures, AR Core does not.

Other advantages of HUAWEI AR Engine enables your device to understand how people move. HUAWEI AR Engine can assist in placing a virtual object or applying special effect on a hand by locating hand locations and recognizing specific gestures. With the depth component, motion tracking capability can track 21 hand skeleton points to implement precise interactive controls and special effect overlays. Regarding body recognition, the capability can track 23 body skeleton points with specific names (Left Hand etc.) to detect human posture in real time. AR Engine supports use with third party applications and the Depth Api. In addition to all these, HUAWEI AR Engine supports these features for both front and back cameras. Also, it is planned to add the feature to provide directions for certain locations in the coming days.

With the AR Engine, HUAWEI mobile phones provide interaction capabilities such as face, gesture, and body recognition, and more than 240 APIs, in addition to the basic motion tracking and environment tracking capabilities.

Differences from HUAWEI ML Kit

HUAWEI AR Engine Body Tracking and ML Kit vision skeleton recognition may look the same. However, there is quite a difference between them. ML Kit provides for some general purpose capabilities while the AR Engine tracks skeleton information in the AR scenario. Skeleton tracking and motion tracking are both enabled in AR Engine. So, AR Engine has various information from the coordinate system. But the service in the ML Kit cannot do the same. Because they served for different purposes.

The service in the AR Engine is used to create an AR app while the service in the ML Kit can only track the skeleton in the image in the smart phone coordinate system. They are different ways to implement these two services. Also, the models are different.

Demo App Development

We will create a simple demo application by using HUAWEI AR Engine’s Body tracking capability. In this demo application, I will try to draw lines that represent the body skeleton on the human body viewed by the camera. First you need to provide software and hardware requirements.

Hardware Requirements

• The current version of HUAWEI AR Engine supports only HUAWEI devices. So, you need a HUAWEI phone that supports HUAWEI AR Engine, can be connected to a computer via a USB cable, and whose camera works properly.(You can see the supported devices in the below table)

Supported Devices

Software Requirements

• Java JDK (1.8 or later).
• Android Studio (3.1 or later).
• Latest HUAWEI AR Engine SDK, which is available on HUAWEI Developers.
• Latest HUAWEI AR Engine APK, which is available in HUAWEI AppGallery and has been installed on the phone.

After providing the requirements, you need to add OpenGL to graphic rendering and the HUAWEI AR Engine SDK dependencies to your app level build.gradle

dependencies{
   //HUAWEI AR Engine SDK dependency
   implementation 'com.huawei.hms:arenginesdk:2.12.0.1'
   //opengl graphics library dependency
   implementation 'de.javagl:obj:0.3.0'
}

To using the camera you need to add camera permission in your AndroidManifest.xml file.

<uses-permission android:name="android.permission.CAMERA" />

Now we are ready for development of our app. Before enter to the development we should know the general process of using the HUAWEI AR Engine SDK. Our demo application AR Engine process has to follow steps in the photo.

Note: The general AR Engine usage process will follow steps in the photo. Development steps are not related with steps in the photo.

Now if you have a look at the general process of AR Engine, we can continue to development.

Note: Remember, if you get confused in these processes, take a look at this picture again after development is complete and you will fully understand the general usage process of HUAWEI AR Engine SDK.

First we need to create a class for body rendering shader utilities. We will use this utility class to create vertex and fragment shaders for body rendering. (We will be take advantage of OpenGL computer graphics library functions to create shaders)

Then we need to create an interface to rendering body AR type related data. We will implement this interface in the classes to be displayed. Then, we will call the overridden methods of the classes that implement this interface from the BodyRenderManager class, which will be created from the onCreate method of our activity.

You can see that we passed ARBody type Collection to onDrawFrame method. There are two reasons for this. First reason is the HUAWEI AR Engine can identify two human bodies at a time by default, and it always returns two body objects. The second reason is that we will draw the body skeleton in overridden onDrawFrame methods. We are using HUAWEI ARBody class because it returns the tracking result during body skeleton tracking, including body skeleton data which will be used on drawing.

Now we need to create display classes to pass the data to OpenGL ES.

The body skeleton display class:

(We will use this class to pass skeleton data to be rendered and displayed on the screen to OpenGL ES)

The body skeleton line display class:

(And this class will be used to pass the skeleton point connection data to OpenGL ES for rendering on the screen)

And the other display class is TextureDisplay class. This class for drawing camera textures that we can use to display camera images on the screen. You can also take a look at the comments in code to understand usage of these class’ methods.

Next, we need to create a class that implements GLSurfaceView.Renderer, a generic renderer interface of OpenGL. GLSurfaceView.Renderer has 3 methods we have to override. These are onSurfaceCreated, onSurfaceChanged and onDrawFrame. We will call init() method of above display classes in the overridden onSurfaceCreated method of this class to create a body skeleton shader on the GL thread. I will explain methods of this class separately. First I will explain onSurfaceCreated method. Because, this method is called at the beginning of rendering.

But, first we should see the constructor of this class

Since, the onSurfaceCreated method of GLSurfaceView.Renderer class is called at the beginning of rendering, as well as every time the EGL context is lost, this method is a convenient place to put code to create resources that need to be created when the rendering starts. Therefore, we should call init() method of above display classes to initialize the objects displayed here.

Now we need to create a class for the demo to adapt to device rotations. This class will be used as device rotation manager. And it should implement Android’s DisplayListener interface. You can find the explanation of each methods in the code.

The onDrawFrame method of BodyRenderManager class called to draw the current frame. This method is responsible for drawing the current frame. So, this is an important method and I’m going to explain this method line by line.

First we need clear the screen to notify the driver not to load pixels of the previous frame.

Note: I’ll add the entire BodyRenderManager class after the explanation.

GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);

Also, we should check whether ARSession is null. If it is null then we can’t continue to drawing process.

if (mSession == null) {
    return;
}

If ARSession isn’t null we have to check whether the current device is rotated. If the device is rotated, we should update the device window of the current ARSession by using updateArSessionDisplayGeometry of DisplayManager class.

if (mDisplayRotationManager.getDeviceRotation()) {
    mDisplayRotationManager.updateArSessionDisplayGeometry(mSession);
}

Set the openGL textureId that used to store camera preview streaming data.
After setting the texture Id, for HUAWEI AR Engine to update the camera preview to the texture ID, we need to call the mSession.update().

mSession.setCameraTextureName(mTextureDisplay.getExternalTextureId());
ARFrame frame = mSession.update();

We obtained the latest ARFrame by the camera with mSession.update(). Now it is time to obtain the projection matrix. First create the array and get the camera specifications of the current frame (ARCamera) by using frame.getCamera(). After that we are able to get projection matrix by using ARCamera.getProjectionMatrix().

Note: You can see the description of each parameter of getProjectionMatrix method here.

// The size of the projection matrix is 4 * 4.
float[] projectionMatrix = new float[16];
ARCamera camera = frame.getCamera();

// Obtain the projection matrix of ARCamera.
camera.getProjectionMatrix(projectionMatrix, PROJECTION_MATRIX_OFFSET, PROJECTION_MATRIX_NEAR,
    PROJECTION_MATRIX_FAR);

Now we can render each frame.

mTextureDisplay.onDrawFrame(frame);

We already rendered texture. Now we will render the body. To get trackable ARBody objects in the ARSession we can use mSession.getAllTrackables (ARBody.class). After obtaining bodies, we will pass bodies and projectionMatrix to the onDrawFrame of display classes to draw bodies.

Collection<ARBody> bodies = mSession.getAllTrackables(ARBody.class);

for (BodyRelatedDisplay bodyRelatedDisplay : mBodyRelatedDisplays) {
    bodyRelatedDisplay.onDrawFrame(bodies, projectionMatrix);
}

You can see the entire of the BodyRenderManager class

And, we should update the texture projection matrix and the viewport rotation when the surface changed. Therefore, we should use methods of the DisplayRotationManager and the TextureDisplay classes in the onSurfaceChanged method of BodyRenderManager class.

Finally we came to the last step of development. In the last step we will use BodyRenderManager class from the our activity. First add a android.opengl.GLSurfaceView to your activity’s layout.

And the onCreate method of our activity. You can see that we just created helper classes like DisplayRotationManager and we made some openGL configurations. At the end of the onCreate method we should check whether HUAWEI AR Engine server (com.huawei.arengine.service) is installed on the current device. I will add entire Activity class and you can see the content of arEngineAbilityCheck method.

I created AR Session in the onResume method. This means that the AR Engine‘s process started from onResume method. Lets take a look at the onResume method.

As you can see that we created ARSession object and we passed it to the setARSession method of BodyRenderManager class. You can check that, in this method we create a new ARSession and the process begins. But, before the passing ARSession to the BodyRenderManager, we create ARBodyTrackingConfig object. This class also provided by HUAWEI AR Engine SDK. When we passed ARBodyTrackingConfig type object to the ARSession.configure(), HUAWEI AR Engine detects the body in the camera viewfinder in real time. After that we enabled DEPTH and MASK capabilities by using ARBodyTrackingConfig.setEnableItem() method. Also we can pass multiple parameters with OR bitwise operator to enable more than one or two capabilities.

Note: Don’t forget the pause, stop and unregister in the related methods like in BodyActivity.

Also, you can see the entire of the BodyActivity class

Now we have done the development steps. You can run the project on your device that supports Huawei AR Engine to test. If you successfully completed all the parts, you should see the application in the below images.

In this post, I tried to explain how to integrate HUAWEI AR Engine into your applications. And I tried to talk about AR Engine advantages based on my research. Thanks for reading so far. I hope this post helps you in developing AR apps.

See you in next posts…

References:

Document