Nvidia Introduces Modular Primitives for High-Performance Differentiable Rendering

Differentiable rendering is a fundamental building block for 3D geometry that enables the gradients of 3D objects to be calculated and propagated through images while also reducing the need for 3D data collection and annotation. In a bid to provide high-performance primitive operations for rasterization-based differentiable rendering, researchers from Nvidia and Aalto University have introduced a modular primitive that uses existing, highly optimized hardware graphics pipelines to deliver performance superior to previous differentiable rendering systems.

The researchers say the proposed modular primitive can implement a more efficient differentiable real-time graphics pipeline, and identify four highlights of the approach:

• Efficiency: Renders 3D scenes that are complex in terms of geometric detail, occlusion, and appearance in high resolution.
• Minimalism: Easily integrates with modern automatic differentiation (AD) frameworks such as PyTorch and Tensorflow.
• Freedom: Supports arbitrary user-specified shading and arbitrary parameterizations of input geometry without committing to specific forms.
• Quality: Supports texture filtering operations with the required mipmap levels generated internally, with no need to make any assumptions about texture content.

Comparison differentiable rendering systems: Scalability refers to surface tessellation and image resolution performance, Flexibility to arbitrary shading support, and Antialiasing to geometric edge smoothing. Occlusion means obscured surfaces won’t affect the resulting image, while Gradients refers to the gradients’ correctness re the rendered image and GI denotes global illumination.

The proposed differentiable rendering systems not only yield good performance, they also provide custom, high-performance implementations such as rasterizing large numbers of triangles, attribute interpolation, filtered texture lookups and user-programmable shading and geometry processing, all in high resolution.

A simple differentiable rendering pipeline with the proposed primitive operations highlighted in red.

Rasterization takes as inputs a tensor of vertex positions (left figure) and a tensor of vertex index triplets (middle figure) that specify the triangles; Interpolation takes the buffer produced by the rasterizer and a buffer specifying the vertex attributes as inputs (right figure).

Texturing provides a per-pixel texture coordinate vector, fetches a value from a texture and places it in the output.

Based on the geometry input (vertex positions and triangles), antialiasing is the last step, which smooths out discontinuities at silhouette edges in a given image.

To assess the performance of the proposed method, the researchers selected meshes of varying triangle counts from the ShapeNet database, then rendered them using both the proposed method and two benchmarks (Soft Rasterizer and PyTorch3D) at multiple resolutions.

In the experiments, the proposed method required much less rendering and gradients time and also showed better scalability. The results also demonstrate that the new approach can achieve excellent geometric correspondence between rendered results and reference imagery.

The paper Modular Primitives for High-Performance Differentiable Rendering is on arXiv.

Analyst: Hecate He | Editor: Michael Sarazen; Yuan Yuan

Synced Report | A Survey of China’s Artificial Intelligence Solutions in Response to the COVID-19 Pandemic — 87 Case Studies from 700+ AI Vendors

This report offers a look at how China has leveraged artificial intelligence technologies in the battle against COVID-19. It is also available on Amazon Kindle. Along with this report, we also introduced a database covering additional 1428 artificial intelligence solutions from 12 pandemic scenarios.

Click here to find more reports from us.

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.