3D content creation for XR from Photo
The attempts to push XR for main stream consumption is evident from the efforts of Facebook (Oculus), Google and many other influencers. While XR gaining traction in both consumer and enterprise space, the demand for 3D content is growing rapidly. Creation of 3D content for XR still remains a challenge. 3D content creation needs sophisticated tools setup with specialised 3D modelling skills. This article outlines the efforts that are made in the public domain to create content for XR from simple photographs.
In this article, we are going to see a process to convert a simple photograph into a usable 3D model for XR. In a nutshell, the entire process flow is as follows
Step 1: Monocular depth prediction
We all know that the photographs are made of pixels. Each pixel representing a RGB color value. In order to generate depth, we need to use a trained ML model and give the whole image as an input. For this article I used a trained monocular depth prediction model “AdelaiDepth” using the Google Colab notebook version of the tool.
Execute the Colab notebook until step “Run Network”. After successfull completion of “Run Network” don’t execute the “Download Results” instead expand the “Files” section and and navigate to “AdelaiDepth/LeReS/test_images/outputs” folder and download the “*-depth.png” file to your local system.
For this sample, My input image and the output depth image looks as follows
Step 2: Create an RGBD image using Open3D
Open a Python shell or IDE and use the following snippet to combine RGB image and depth image into one RGBD representation
import open3d as o3dcolor_raw = o3d.io.read_image("interior_deco.jpg")depth_raw = o3d.io.read_image("interior_deco-depth.png")rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(color_raw, depth_raw)print(rgbd_image)
Depending upon the image dimension, the output may vary. You may see some output like below
Step 3: Convert the RGBD image as 3D Point cloud
Continue evolving the script written in step 2 and convert the RGBD image created into a point cloud using Open3d’s PointCloud package
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd_image,o3d.camera.PinholeCameraIntrinsic(o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))# Flip it, otherwise the pointcloud will be upside downpcd.transform([[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])o3d.io.write_point_cloud("interior_deco.ply",pcd,True)o3d.visualization.draw_geometries([pcd])
Save the generated point cloud in PLY format using “write_point_cloud” function as shown above. This PLY we will use it in the next steps to create the 3D mesh.
Step 4: Cleanup Pointcloud
Import the PLY file created in the previous step in “MeshLab”. Scale it up 100 times by right clicking on the layer panel and selecting “Matrix: Set from translation/rotation/scale”
The sample PLY used in this article has 192,500 vertices for this simple scene. For XR applications, the mesh need to have very optimal number of pixels in order to get a decent FPS during the XR experience. So lets reduce them by using the Meshlab’s point cloud reduction function “Filters → Remeshing … → Simplification: Clustering Decimation”
Leave the default values and click apply. Clustering decimation gets rid of redundant points in the point cloud and gives us a minimal and optimal set of points.
After simplifying the point cloud, select the newly created point cloud from the right side panel and compute normals. Normals are needed for meshing operation.
Step 5: Create 3D Mesh from Pointcloud
After creating the normals, Perform “Filters → Remeshing … → Surface Reconstruction: Screened Poisson”
Take the default values and click apply. You will get a poisson mesh.
Poisson meshing may introduce additional face at the boundary, which can be cleaned up by selecting faces greater than a specified length and delete them.
Adjust the Threshold value for optimal selection and click apply. Then “Filters → Cleaning and Repairing → Delete Selected Faces and Vertices”
Step 7: Export Mesh
Now the mesh can be exported using “File → Export Mesh”
If you are planning use the 3D mesh in XR development environment like Unity, then select “.Obj” as target format.
There you go. You have successfully converted a Photo into a 3D model that can be used in XR. The quality of the mesh may appear rough today. However, as the precision of the Monocular depth sensing improves in the coming years. The quality of the mesh will improve as well.
The process of automating the Meshlab steps provided in this article is parked for future :-)
PS: Parts of this article was written while being quarantined. When the force is with you, You stop at nothing :-)
