Mastering the art of AI-generated XR content: An improved approach
Alright folks, are you ready for round two? Because I am bringing you the sequel to our last article on how to turn text into 3D XR content. If you haven’t read the first part, I suggest you go back and give it a read before diving into this one. But for those of you who have, buckle up because things are about to get even more interesting.
In the previous article, we discussed a method for converting text into 3D content for use in XR prototyping. While the results were promising, there were two main issues that needed to be addressed to make the final output usable.
Firstly, the detected depth did not accurately reflect the true depth of the image.
Secondly, the meshing and coloring of the model was not visually pleasing.
In this article, we will explore ways to address these issues and improve the overall quality of the final output.
Recovering 3D Shapes from Single Image:
To improve the depth detection, we will be using a different model produced by the research paper “Learning to Recover 3D Scene Shape from a Single Image”. This model is available through a Git Repository, and we found that it produced a more accurate depth prediction. By using this model and following the point cloud generation process outlined in the previous article, we were able to achieve a more usable final outcome in the point cloud.
Next in order to get a better looking mesh which can be used in AR/VR environment, we need the following workflow
1. Normals Generation → 2. Poisson Mesh → 3. Color to Texture → 4. Export
First, we generated normals for the point cloud using the “Filters → Normals, Curvatures and Orientation → Compute normals for point sets” option. Here, we set the neighbours to 30 and checked the “flip normals” option.
If your lightings are set correct, you will already see good 3D output emerging from the point cloud. Next we run screened poisson with reconstruction depth set to 10.
Once the mesh got constructed, Run Filters → Remeshing … → Simplification: Quadratic Edge Collapse Decimation to reduce the number of points and triangles in the mesh to make it AR friendly.
Set the target number of faces to AR friendly numbers. I prefer 10k to 30k. Also check Preserve Boundary of the Mesh.
After that, we will use the Trivial Per-Triangle filter under Filters > Texture > Parameterization to assign a unique texture to each triangle in the mesh. We will set the texture dimension to 4096 and the method to “Basic”.
Finally, we will use the Vertex Attributes to Texture filter under Filters > Texture > Transfer to transfer the color information from the original point cloud to our Poisson mesh. We will set the texture width and height to 4096 and provide the original point cloud and Poisson mesh as the source and target meshes, respectively.
Now you will have an AR ready model as shown below
This model is finally ready to be exported for AR/VR. The entire flow discussed in the article can be automated as a workflow toolchain for cloud.
