Meshes data preparation for 3D-Segmentation

4 min readMay 11, 2023

There are many techniques had been shared by others regarding 3D segmentations. Main use case mostly in healthcare for medical diagnosis.

However there are 2 view points that rarely being discuss:

Data preparation
Utilisation of this techniques in wider industries

In this article, the main topic will be data preparation:

The journey to understand 3D segmentation started with obtaining 3D data, Shapenet is the main source of data for this purposes. The model to be used is car model.

The car model that available then being labelled using FreeCAD.

All the 3D data is represented by obj file (“s a geometry definition file format first developed by Wavefront Technologies for its Advanced Visualizer animation package. The file format is open and has been adopted by other 3D graphics application vendors”)..

With all the segmented data, the intention is to train using MeshCNN or CNN based 3D segmentation.

The problem here is, the segmented data consist of hundred of thousands of vertices and faces which took huge compute power for model training. In order to start small, there’s a need to downsize this model.

There are various ways to downsize the vertices and faces, 1 solution that worked is downsizing with Vertex Clustering (“The vertex clustering method pools all vertices that fall into a voxel of a given size to a single vertex.”)

Downsize each segmented car model using Open3D simplify_vertex_clustering

import open3d as o3d

ori_filename = 'car.obj'
ori_mesh = o3d.io.read_triangle_mesh(mesh_filename)
ori_mesh.compute_vertex_normals()

dv_size = max(ori_mesh.get_max_bound() - ori_mesh.get_min_bound()) / 32
down_mesh = ori_mesh.simplify_vertex_clustering(
          voxel_size=dv_size,
          contraction=o3d.geometry.SimplificationContraction.Average)

o3d.io.write_triangle_mesh('car_downsize.obj', down_mesh)
o3d.visualization.draw_geometries([down_mesh])

Once the segmented image had been downsize it then can be merged back as one model as below.

seg_mesh = o3d.io.read_triangle_mesh(segmented_mesh_filename)
if merge_mesh == None:
    merge_mesh = seg_mesh
else:
    merge_mesh = merge_mesh + seg_mesh

Generate label file: Once downsize car meshes and segmented car meshes generated, the labelling for each vortex can be done with below code.

import open3d as o3d

import glob
import os
import numpy as np

def string_to_number(argument):
    switcher = {
        "body": 0,
        "boot": 1,
        "bumper_front": 2,
        "bumper_rear": 3,
        "doors": 4,
        "fenders": 5,
        "glass_pc": 6,
        "hood": 7,
        "others": 8,
        "roof_metal": 9,
        "wheels": 10,
        "boot_mould": 11,
        "spoiler":12,
        "roof_glass":13,
    }

    return switcher.get(argument, "nothing")

def generate_labels():
    #Get all the original meshes for simplification
    path_to_label = "/datasets/car_model/merged_label/"
    #Get all the downsized and merged meshes for labelling 
    path_to_meshes = "/datasets/car_model/merged/*.obj"
    downsize_mesh_filenames = glob.glob(path_to_meshes)
    
    path_to_segmented_meshes = "/datasets/car_model/segmented_downsize/"

    #loop the car images that had been downsize and merged as one full car
    for down_mesh_filename in sorted(downsize_mesh_filenames):

        with open(down_mesh_filename, 'r') as f:
            data = f.readlines()

            # Extract vertex and face data from obj file
            vertices = []
            for line in data:
                if line.startswith('v '):
                    vertex = line.split()[1:]
                    vertices.append(vertex)


        _filename = os.path.basename(down_mesh_filename).replace('-downsize-merged.obj','')
        segmented_meshes_filenames = glob.glob(path_to_segmented_meshes + _filename + "*.obj")
        
        t_vseg = [] 
        vseg = np.zeros((len(vertices),), dtype=int)
        #loop segmented car meshes to match the vertices and generate a file
  
        for segmented_mesh_filename in sorted(segmented_meshes_filenames):
            print(segmented_mesh_filename)
            with open(segmented_mesh_filename, 'r') as sf:
                seg_data = sf.readlines()
                seg_vertices = []
                for seg_line in seg_data:
                    if seg_line.startswith('v '):
                        seg_vertex = seg_line.split()[1:]
                        seg_vertices.append(seg_vertex)
    
            seg_filename = os.path.basename(segmented_mesh_filename).replace(_filename + '-','').replace('-downsize.obj','')
            
            
            for i, vertice in enumerate(vertices):
                for j, svertice in enumerate(seg_vertices):
                    if np.any(np.all(vertice == svertice)):
                        vseg[i] = string_to_number(seg_filename) 
                        break
        for i, vertice in enumerate(vertices):
            if vseg[i] == None:
                vseg[i] = 0
                
        with open(path_to_label+os.path.basename(_filename)+'.text', 'w') as vp:
            for ver in vseg:
                vp.write("%s\n" % ver)

Now the data and label is ready to be used for 3D model segmentation based on meshes vertices.

References

[1] Shapenet https://shapenet.org/.

[2] Wavefront (.obj) https://en.wikipedia.org/wiki/Wavefront_.obj_file

[3] Open3D http://www.open3d.org/docs/release/tutorial/geometry/mesh.html#Vertex-clustering

Meshes data preparation for 3D-Segmentation

References

Written by Santi Ulung