Point Cloud 3D pipeline for Unity using Python — Part 1
3D content generation and processing is a challenge for XR application development. Traditional 3D content pipelines need a 3D artist with good hands-on experience with modeling tools such as Blender or the commercial ones like Maya, 3D Studio, and others. This is a time consuming and not accessible/affordable to small teams and indie developers.
Alternate in producing a 3D model is to scan a real-world object using a 3D scanner like LiDAR and other similar hardware or use a photogrammetry solution such as “Visual SFM” and have it convert a sequence of photographs into a 3D model. In both cases, the output is a 3D point cloud.
Unity has recently released an alpha preview of its plug-in “Python For Unity”. Prior to this plugin, the only way to convert the point cloud is to use external software tools like “Mesh Lab” to mesh the point cloud and have it converted into a 3D object. While the meshlab process shall be configured with templates, it doesn’t offer a programmatic control over the conversion process.
In this article, we are going to outline a simple process to convert point clouds into 3D models within Unity Editor using Unity’s Python API and Open3D python library. Open3d is an open-source library that supports the rapid development of software that deals with 3D data. The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python.
Note: At the time of writing this article, The Python for Unity plugin is an alpha release and supports only Python runtime version 2.7. Hence the output may be far from putting it to any practical use. However, this will get you started with the process and will open you with umpteen number of possibilities with Python and the packages available with python out there.
Step 1: Installing Open3D library
Assuming you have already installed Python 2.7 and Conda. Create a new Conda environment with python 2.7 as target and activate it.
conda create -n newenv2.7 python=2.7
conda activate newenv2.7Install open3d package using Conda. At the time of writing this article, only open3d v0.8.0.0 works well with Unity 2019.3 and Mac OS 10.14. To install this specific version.
conda install -c open3d-admin open3d==0.8.0.0also, install lru_cache package incase if the “import open3d” statement fails.
conda install backports.functools_lru_cacheStep 2: Create Unity Project & add Python for unity package
The steps in this article are written for and tested using Unity 2019.3. If you are using a different version, check the compatibility of “Python for Unity”.
a. Create a new 3D project in unity hub.
b. Using a text editor, open the <ProjectPath>/Packages/manifest.json and add “com.unity.scripting.python”: “2.0.1-preview.2” to the dependencies as shown below. Save the file.
{"dependencies": {"com.unity.scripting.python": "2.0.1-preview.2","com.unity.collab-proxy": "1.2.16",
c. Open the project in Unity Editor, Goto “Edit -> Project Settings -> Python for Unity”
d. Add the package directory in the Site Packages section as shown in the above image.
e. Restart Unity and open the same project
Step 3: Executing a Python script within Unity
For demonstrating a point cloud meshing, The LiDAR 3d scan of motorcycle from sketch fab is used. Download the “.obj” model from here. Copy the obj file containing the point cloud under “<Project Folder>/data/”.
In Unity, Goto “Window -> General -> Python Console”
Copy / Paste the following code in the Python Script editor window (bottom half),
import os
import sys
import numpy as np
import open3d as o3d
import UnityEnginefilename = os.path.abspath('data/head-pointcloud.ply')
output_path = os.path.abspath('Assets/model.obj')
UnityEngine.Debug.Log(filename)pcd = o3d.io.read_point_cloud(filename)
downpcd = pcd.voxel_down_sample(voxel_size=0.03)
distances = downpcd.compute_nearest_neighbor_distance()
avg_dist = np.mean(distances)
radius = 3 * avg_distbpa_mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_ball_pivoting(downpcd,o3d.utility.DoubleVector([radius, radius * 2]))
bpa_mesh = bpa_mesh.filter_smooth_laplacian()
o3d.io.write_triangle_mesh(output_path, bpa_mesh)
UnityEngine.Debug.Log(output_path)
This python script opens the point cloud, downsamples the point cloud using a tolerance voxel size and constructs a mesh using a Ball pivoting algorithm, and saves the output mesh to the Assets folder which will be auto imported by Unity.
Input Point Cloud:
Output Mesh:
Summary:
While the output is still not production-ready, The choice of having the ability to use and integrate python into Unity’s asset pipeline is a welcome move. It would be good if the official release of Python for Unity has support for Python 3 and above also has a better integration as In-Process API. This would already enable us to write asset processing toolboxes for Unity and Point Cloud toolbox could definitely be one of them.
