Beginner: Object Reconstruction application using Huawei 3D Modeling Kit

Object Reconstruction using Huawei 3D Modeling Kit

3D Modeling Kit is an AI-powered kit automatically generates 3D models and physically based rendering (PBR) texture maps, to satisfy the needs for efficient 3D model and animation creation.

Introduction

In the rapidly advancing world of technology, 3D Modeling has become an integral part of various industries, from gaming to architecture and manufacturing. To empower developers in creating immersive and visually stunning applications, Huawei Mobile Services (HMS) has introduced 3D Modeling Kit. The 3D model plays an important role in boosting user conversion. It allows users to carefully view an object from every angle before they make a purchase or decision.

3D Modeling Overview

Huawei 3D Modeling Kit provides the AR Capability to get the image rendered with various features from local and cloud. This helps the users to develop the application with low cost to make 3D content.

Integrate this capability into your app for image data collection and upload, model download, as well as 3D object preview, enabling your users to construct 3D object models from images even on mobile phones without hardware such as RGB-D or light detection and ranging (LiDAR) sensors. 3D Modeling Kit provides four capabilities to help with creating 3D content more efficiently at a lower cost.

• Material generation: Integrate this capability into your app to enable your users to convert images into five types of texture maps: diffuse map, normal map, specular map, roughness map, and height map.
• 3D object reconstruction: Integrate this capability into your app for image data collection, data upload, and model download, as well as model preview and extra scanning, enabling your users to construct 3D object models from images even on mobile phones without hardware such as RGB-D or light detection and ranging (LiDAR) sensors.
• Motion capture: Use this capability that can load a preset motion to a 3D model of a biped humanoid by using the skeleton points on the model. In this way, the capability automatically rigs and animates a biped humanoid model, to lower the threshold of 3D animation creation and make 3D models appear more interesting.
• Auto rigging: Use this capability to output 3D data (quaternions and 3D coordinates) of 24 key skeleton points of the human body in a 2D video.

In addition to the capabilities above, 3D Modeling Kit also provides a library of various materials that you can use as needed. This library supports material query, preview, and download, helping improve your work efficiency.

Here in this article we will go deep into how to do 3D Object Reconstruction using HMS 3D Modeling Kit.

3D Object Reconstruction — Features

• Automatically generates a texturized 3D model for an object by using images captured from multiple angles. The following modeling modes are available: common mode and real-time guide mode.
• Requires longer time for modeling an object whose image resolution is higher.
• Does not allow for modeling for the human body or face.

There are 2 modes available.

1. Common Mode
   - Supports the following object types: goods (including plush toys, bags, and shoes), furniture (like sofas), and cultural relics (such as bronzes, stone artifacts, and wooden artifacts). Objects must feature detailed textures, and not be reflective, transparent, semi-transparent, or purely black.
   - Supports the following object dimensions: between 10 x 10 x 10 cm and 2 x 2 x 2 m.
   - Allows for extra scanning of a modeling task. Two times of extra scanning are supported. The second extra scanning request can be sent only after the first extra scanning task delivers a modeling result (success or failure). The second extra scanning task can be based on either the original modeling task or the first extra scanning task with a successful modeling result.
   - Allows for configuring the mesh count level to high (100,000 meshes), medium (50,000 meshes), or low (25,000 meshes).
2. Real-time Guided Mode
   - Supports the following object types: goods (including plush toys, bags, and shoes) and cultural relics (such as bronzes, stone artifacts, and wooden artifacts). Objects must feature detailed textures, and not be reflective, transparent, semi-transparent, or purely black.
   - Provides an AR hemisphere bounding box (which serves as the guide for taking images from different angles for an object) and automatically generates a texturized model for this object.
   - Requires the object dimensions be less than 30 x 30 x 30 cm.
   - Supports the following image resolutions: 1440 x 1080 px (on Huawei phones which use AR Engine) and 480p to 1080p (on non-Huawei Android phones).
   - Requires Android devices that support AR Engine (which is supported by default on Huawei devices) or ARCore. The real-time guide mode depends on the AR Engine SDK (for Huawei phones to implement this mode) or the ARCore SDK (for non-Huawei phones to implement this mode). Therefore, the mode is unavailable for an app without the above SDKs.

Development process

It is assumed that the developer has prior knowledge on Android Application Development.

Hardware Requirement

• A computer (desktop or laptop) running Windows 10.
• Huawei Android phone (with the USB cable), which is used for debugging.

Software Requirement

• JDK 1.8 or later
• Android 7.0 or later
• Android Studio 3.6 or later
• Gradle 5.0 or later

Preparations

Before you start developing an app, you must complete the following preparations:

1. Register as a Huawei developer account in AppGallery Connect.
2. Create an app in Android Studio IDE.
3. Generate Signing Certificate Fingerprint and configuring the information in App Gallery Connect.
4. Enable Data processing Location and download the AGConnect-Services.json file add it under the android/app folder.
5. You need to add Huawei 3D Modeling Kit to your app.

Kindly look into this link for detailed instruction.

Steps to create Android Project

1. Open Android Studio.
2. Click NEW Project, select a Project Template.
3. Provide New project name and Package Name.
4. Select Java from the Language drop-down list, and set the Minimum SDK to API 24: Android 7.1 and click Finish.

Modifying Configuration Parameters

• Open the build.gradle file in the root directory of your Android Studio project.

• Add the Maven Repository details in the file as mentioned below in the buildscript > repositories and allprojects > repositories.

• Add the following configuration in settings.gradle file

• Add the 3D Modeling Kit dependency and the AppGallery plugin in the app-level build.gradle file available in android/app directory.

Development

Step 1: Add necessary permission in AndroidManifeast.xml file.

Step 2: In the OnCreate method Configure the authentication information for your app.

Step 3: Inititalize the material3D texture engine in Capture activity.

Step 4: Upload listener created to process the image upload result.

Step 5: Now upload the collected image to cloud as shown in the figure.

Step 6: In an Adaptor class query the material generated task. On the download button click initialzing and set the queryTask().

Step 7: Now create a callback for the listener to process the generated texture.

Step 8: Download the generated texture file and call the Synchronus API to obtain the generated texture.

Step 9: Run the application.

Output

Tips and Tricks

• Always use the latest version of the library.
• Make sure you have properly enabled all required services in AppGallery Connect.
• Makes sure that agconnect-services.json file is added.
• Generating SHA-256 certificate fingerprint in android studio and configure in Ag-connect.
• Make sure dependencies are added in build files.
• Make sure proper permissions are provided in AndroidManifest.xml

Conclusion

This article will help you to learn the integration of Huawei 3D Modeling Kit and one of its features 3D Object Reconstructing into your application.

Thank you for reading and if you have enjoyed this article, I would suggest you to implement this and provide your experience.

References

3D Modeling Kit

Document

华为开发者学堂