Rendering Volumes and Implicit Shapes in PyTorch3D

PyTorch
PyTorch
Feb 24 · 3 min read

Authors: David Novotny, Roman Shapovalov, Nikhila Ravi, Shubham Goel, Georgia Gkioxari, Justin Johnson, Jeremy Reizenstein, Patrick Labatut, Wan-Yen Lo

Our logo is an output of PyTorch3D’s implicit volume renderer

Intro

PyTorch3D is a highly modular and optimized library with unique capabilities designed to facilitate 3D deep learning with PyTorch. PyTorch3D provides a set of frequently used 3D operators and loss functions for 3D data that are fast and differentiable, as well as a modular differentiable rendering API. Researchers can use these features in deep learning systems right away. (For a primer on PyTorch3D and differentiable rendering have a look at our tutorial at the PyTorch hackathon).

Implicit Shape Rendering

Today we are releasing a suite of new features to support implicit shape rendering. In the past year there’s been an explosion in the number of papers and projects that employ neural rendering. This exciting research direction focuses on generating realistic renderings of 3D scenes from novel viewpoints based on input scene images. The core idea — reconstructing the implicit representation of surfaces in a 3D scene with a neural network coupled with differentiable rendering — enables learning the geometry of the 3D scene only from 2D views.

Efficient research in this direction requires several key engineering components including an abstraction for volume data as well as a differentiable implicit shape renderer. To enable flexible experimentation in this nascent research area, we provide a modular and extensible API. We have identified the key reusable components and provide well documented and tested implementations of them.

The new features we have added to PyTorch3D include:

  • a Volumes data structure to support batching of 3D volumes and conversion between coordinate frames
  • multiple ray sampling implementations (GridRaysampler, MonteCarloRaysampler, NDCGridRaysampler)
  • several raymarcher implementations (AbsorptionOnlyRaymarcher, EmissionAbsorptionRaymarcher)
  • ImplicitRenderer and VolumeRenderer APIs which compose a Raysampler and a Raymarcher
  • several utility functions such as differentiable conversion of point clouds to volumes

Getting Started

To showcase the new components, we are including a modular, well documented reimplementation of NeRF using these components. Our re-implementation runs faster than the official release, while matching the quality of the output images. Below is a sample of a scene with many shapes and complex reflections generated with the PyTorch3D-based implementation of NeRF.:

Credit: Leonid Naku [Zurel] 2016

This code can be used as a starting point for any research project in novel view synthesis. It will live in the new `projects` folder in the PyTorch3D repo where we will continue to add more examples of projects and paper implementations using PyTorch3D features.

To learn more about these techniques, see our new Colab tutorials that will take you step by step through examples of fitting a textured volume a simple NeRF model.

Conclusion

The goal of PyTorch3D is to drive progress at the intersection of deep learning and 3D by equipping researchers and engineers with a toolkit to implement cutting-edge research with complex 3D data. We commit to continuously improve and expand on the operators in PyTorch3D and welcome contributions from the community.

For a more detailed explanation of the new PyTorch3D tools for rendering implicit shapes and volumetric voxel grids, check out this below video tutorial:

Facebook AI Research Scientist David Novotny gives an overview of the latest release of PyTorch3D. The release comprises a set of tools for rendering and processing implicit/volumetric shapes to facilitate state-of-the-art 3D deep learning research.
PyTorch

An open source machine learning framework that accelerates the path from research prototyping to production deployment

PyTorch

Written by

PyTorch

PyTorch is an open source machine learning platform that provides a seamless path from research prototyping to production deployment.

PyTorch

An open source machine learning framework that accelerates the path from research prototyping to production deployment