Learning3D: A Modern Library for Deep Learning on 3D Point Clouds Data
Learning3D is an open-source library that supports the development of deep learning algorithms that deal with 3D data. The Learning3D exposes a set of state of art deep neural networks in python. The code is clean and consistently styled. A modular code has been provided for further development. We welcome contributions from the open-source community.
[Note: Author of this library is constantly working to improve it. Any suggestions/recommendations are always welcome. (Contact)]
Core Features:
- Basic neural networks that process point clouds
- Well written and consistently styled network structures.
- Basic data-loaders for various tasks (classification, segmentation, registration, reconstruction, etc.)
- Pre-trained models with ModelNet40 data set.
Supported Systems:
- GCC 5.4 and higher version
- Ubuntu 16.04 and higher
- Pytorch 1.3 and higher
- Open3D 0.7
- torchvision 0.4.1
- CUDA 10.0 or higher
Available Networks in this Library:
- Point Feature Embedding Networks: PointNet, DGCNN
- Registration Networks: PointNetLK, PCRNet, DCP, PRNet
- Flow Estimation Networks: FlowNet3D
- Reconstruction Networks: Point Completion Network (PCN)
Installation:
- git clone https://github.com/vinits5/learning3d.git
- cd learning3d
- pip install -r requirements.txt
Documentation:
The documentation of all the important classes, functions and their parameters are described on our GitHub.
Basics:
Let’s look at a very easy way to write a point cloud classifier.
As you can see, it is fairly easy to train a PointNet classifier network for 3D point clouds on ModelNet40 data with the use of Learning3D library. Now, we will look at the above code in details.
- First section imports all the necessary modules from the Learning3D & Pytorch.
- Second section of code creates a PointNet embedding network with 1024 dimensional features for each point and includes batch normalization. This network has been used with Classifier which has Fully Connected Layers to compute the probabilities for each class.
- Third section creates batches of ModelNet40 to train our classifier network with 32 batch size and randomly shuffling the point clouds after each epoch.
- Fourth section defines an optimizer to minimize the cross entropy loss function for classification.
- Fifth section has the actual training loop where each batch of point clouds are classified by network and then these predictions (output) and ground truth labels (target) are used to optimize the weights of the classifier.
Testing a point cloud classifier.
In the above sections, we looked at the training and testing of basic point cloud classifier. Learning3D will allow you to write easy and clean codes for training and evaluation of various deep neural networks associated with 3D geometry / point clouds.
All such sample training and testing codes are available here. (in the examples directory of Learning3D library on GitHub)
Advance:
Along with networks doing classification, segmentation, etc., Learning3D library has a suite of networks that 3D point cloud registration. Few of these methods are correspondence less methods (like PointNetLK, PCRNet) while others rely on explicit correspondence estimation (like DCP).
As we provide pretrained models of these registration networks, I would also like to show their performance.
All these networks are trained using Learning3D library and pre-trained models are provided with it. These statistical results are generated using 1000 pairs of source and template point clouds for each value of initial misalignment (shown on x-axis). Also, these point clouds are randomly chosen from the ModelNet40 dataset.
Above results show that PointNetLK gives best results among all when the initial misalignment is below 45 degrees. When it increases above 45 degrees then, there is a drop in PointNetLK’s performance. But, DCP has shown promising results even at higher initial mis-alignments. This shows that all these methods have their own pros and cons.
Contributions to Learning3D
The master branch is used only for stable development versions of Learning3D. Any code change is made through four steps using the issues and pull requests system.
- An issue is opened for a feature request or a bug fix.
- A contributor starts a new branch or forks the repository, makes changes, and submits a pull request.
[Note: Refer this link to make a clean pull request.] - Code change is reviewed and discussed in the pull request. Modifications are made to address the issues raised in the discussion.
- Admin will merge the pull request to the master branch.
References:
I would like to thank following researchers and their articles.
- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
- Dynamic Graph CNN for Learning on Point Clouds
- PointNetLK: Robust & Efficient Point Cloud Registration using PointNet
- PCRNet: Point Cloud Registration Network using PointNet Encoding
- Deep Closest Point: Learning Representations for Point Cloud Registration
- PRNet: Self-Supervised Learning for Partial-to-Partial Registration
- FlowNet3D: Learning Scene Flow in 3D Point Clouds
- PCN: Point Completion Network
- 3D ShapeNets: A Deep Representation for Volumetric Shapes
Upcoming Features
- Point-wise Rotation Invariant Network (PRIN)
- PointSIFT: A SIFT-like Network Module for 3D Point Cloud Semantic Segmentation
- PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D local descriptors
- Just Go with the Flow: Self-Supervised Scene Flow Estimation
- AlignNet-3D: Fast Point Cloud Registration of Partially Observable Objects
License
Learning3D is released under the MIT license. We encourage use for both research and commercial purposes, as long as proper attribution is given. Feel free to also send us an email and let us know how Learning3D has been useful to you and how it can be improved.