How to build a sport app powered by Smart Device Kits

“Smart Device” Kits are part of Huawei HMS Core

After the first burst of innovation, the landscape of mobile app development has not changed that much during the last years. Despite the number of smart devices every person uses is growing quickly, the apps and services that we use daily are still struggling to share content across different devices in a seamless way.
Huawei’s strategy is to build an ecosystem of connected devices and related technologies to let developers build apps that are not confined anymore in the hardware limitation of a smartphone, but can be extended to create a smarter environment around the user.

Imagine if you can build an app that is not only designed for your phone, but also for your home, your car or your gaming or sport equipment!

As an interesting example, I’d like to show you how to build a sport app that take advantage of smart devices such as external cameras and smartwatches.

Goal

Practically, let’s build an Android app that is able to:

• Discover smart devices around you
• Connect to the action camera and the smartwatch
• Get live stream video from the camera
• Get health data from the smartwatch
• Overlay health data to the video stream
• Enjoy the video :)

Preparation

To help developers building such scenarios, Huawei is providing several powerful libraries. In this project we are going to use 2 of them in particular: Device Virtualization (aka DV Kit) and HiHealth Kit!

The goal of DV Kit is to provide a single framework for a seamless multi-device interaction. There are 2 sets of libraries, one to be integrated in the Android app and another one in the Smart Device.

On the app side, the advantage is that developers don’t need to write specific code to interface with external devices, but those are accessible in a similar way as internal resources!

HUAWEI HiHealth is an open platform facilitating easier data access and service aggregation of biometrics data, and consists of the HUAWEI Health app and HUAWEI HiHealth cloud.

The HUAWEI Health app is an official app from Huawei and comes pre-installed on all Huawei phones, providing users with diverse functions, such as workout data tracking, advanced fitness training, sleep monitoring, and healthy lifestyle services.

What you will need

• A Huawei smartphone equipped with Kirin 980 or later and running EMUI 10.1+ (at the time of the article I recommend Huawei P40)
• A Huawei smartwatch (I am using Huawei GT Watch Active 2)
• An action camera compatible with DV Kit (for example Drift Ghost 4K+)
• Love for sport and new technologies!

Implementation

Security first, let’s apply for permissions.
DV kit is part of the family of platforms which are exclusively for businesses and hence any app has to be approved before it can access the APIs.

That’s why as a first step you need to register as a developer in Huawei’s portal.
From the console then submit an app, upload your signing certificate fingerprint and and keep note of your app id. For more detailed steps see environment preparation.

Create a new project and add custom permissions in the manifest:

Those permissions are checked at runtime and verified by Huawei Cloud, for this reason the app must be signed with the same certificate previously uploaded in the console.
Also, the app id must be specified in the manifest:

Some Android standard permissions are also needed to access the Camera API, to record audio and to access body sensors data:

Now it’s time to include DV kit dependencies in the gradle file. Add the repository in both buildscript and allprojects locations:

While in the app-level gradle file, configure the build dependencies:

You can find all the sdk versions published here.

HiHealth Kit comes in the form of a jar file that you need to manually download from the developer website and add in the libs folder.
For exploration purposes, you can use the debug version, but in order to publish an app that uses HiHealth APIs you firstly need to apply for those permissions, similarly to DV kit.
If you want to join the developer program see HiHealth data access introduction.
Once you got the artifact, add the gradle dependencies in this way:

Integrating Device Virtualization

In an over-simplified scenario we will need just 2 objects, a VirtualDeviceManager that handles the lifecycle of the service and a collection of VirtualDevice objects to keep track of the smart devices around the smartphone.

For simplicity I will use an HashMap where the key is the device id. It can be declared like this:

Then let’s initialize the DVKit service, calling the following function in the onCreate phase:

You may notice that we need to define 2 additional objects to specify what happens when a new device gets discovered and when a new capability is enabled.
In the first case, we just want to add the discovered device to our collection (ideally avoiding duplicates), we can do that by implementing the IDiscoveryCallback:

In the second case, the IVirtualDeviceObserver will notify that a new capability has been detected. In particular, we want to use the just discovered smart camera, enable it and start a new Activity passing the camera id as parameter.

That’s it! Now the smartphone is connected with the external camera and we can work with it just like we would do with the phone’s internal one.
Let’s see how.

For simplicity, we will use the camera in a separate Activity, an interesting use case is to put the phone in the pocket once it is connected to the external camera (mounted on the helmet or bike) while keeping the live stream ongoing. In this case it’s better to use a Service instead, the code is anyway very similar.

As already said, once the external camera has been connected, it can be used just like any other camera, consider it a CameraDevice object from Android Camera 2 API.

The Activity got the cameraId as a parameter, another property that will be useful is the range of available frames target, keeping this in mind, the function to open the camera will look like this:

The stateCallback needs to be initialized:

Now, the tough part. This article is not meant to be an exhaustive guide on how to work with cameras in Android, many things can be done and my suggestion is to give a look at the official documentation.
However it’s important to say that we need to define a CaptureRequest to be used in a CameraCaptureSession.
So the startPreview(cameraDevice) looks like this:

Since we want to add an additional layer with biometrics data on the top of the video stream, we are going to use a custom SurfaceView which also needs to be an OpenGL Render. Let’s call this class OverlayView and let’s explore its main characteristics.
From the previously defined startPreview(cameraDevice) function we know that the class holds a Surface, images drawn to this Surface can be attached to an OpenGL ES texture.
Without going too deep in the OpenGL ES functionality, let’s assume we have a draw(data: BiometricData) function that simply draws a Sprite icon and a Text on the surface.

Now It’s time to retrieve this real time data from the smartwatch!

Integrating HiHealth

We are now going to use the Java APIs to retrieve data from the HUAWEI Health app, which can be synchronized in real time with the smartwatch.

User needs to grant permissions to let third-party apps accessing health data by accepting a dialog box. The dialog can be shown using the requestAuthorization(context, writingPermissions, readingPermissions,listener) method from HiHealthAuth class.
In our case, a simple method would be this:

In particular, we are asking the user to let our app accessing the heart rate and other real time sport data. All the possible read/write permissions are listed here.

As a starting point, let’s see how to read heart rate information. Additional data can be retrieved with a similar approach.
It’s time to meet our special guest, the HiHealthDataStore. This object is the main interface between our app and user’s data.

It has 2 convenient method that we are going to use: startReadingHeartRate and stopReadingHeartRate.
The results are in JSON format, it’s common to define a data class to store the information and to take advantage of the Gson deserializer:

Getting the desired data now looks like this:

The heartBeatRate value can be consumed by an observer that will notify our OverlayView, which will eventually draw the data on screen.
Another interesting method is startRealTimeSportData: it can be used in a similar way with an HiSportDataCallback to obtain a Bundle of data, including speed, calories burnt, altitude and many others. A complete list of all the keywords can be found here.

Conclusions and next steps

This is an end to this article but beginning of our journey as there will be more content on how to build innovative multi-device use cases using our HMS Kits!

If you want to see a wonderful example you can check the Drift Life App, which fully leverages DV Kit and HiHealth to bring an innovative experience to their users.

By the way, this is just one of the possible scenarios that can be realized using newest HMS Smart Device kits. Huawei Smart Device kits are in active development and more and more scenarios will be supported in future. I would like to hear your ideas and feature requests!

A further investigation could be the real-time processing of the video stream using HiAI: Huawei’s powerful tool to run neural networks on device. If the smartphone is going to be the center of a connected smart enviroment, better to learn how to fully leverage the potential of its “brain”!

I hope to have intrigued you with this article, if you are interested enough to try developing an app that uses these technologies, do not hesitate to contact me or ask in the comment section!