Generate a Complete 3D Scene Under Arbitrary Lighting Conditions from a Set of Input Images

This new method is able to generate a complete 3-dimensional scene and has the ability to decide the lighting of the scene. All this with very limited computation costs and amazing results compared to previous approaches.

Image via: P. P. Srinivasan et al., “Nerv: Neural reflectance and visibility fields for relighting and view synthesis”

NeRV, or Neural Reflectance and Visibility Fields for Relighting and View Synthesis, is a method that produces a 3D representation of a scene and can generate arbitrary lighting conditions. It only needs a set of images of the scene as inputs to generate novel viewpoints of the scene under any chosen lighting conditions!

Image via: P. P. Srinivasan et al., “Nerv: Neural reflectance and visibility fields for relighting and view synthesis”

This is an extremely complicated task since to simulate the lighting effect of a scene, we need to compute the visibility of each point with each light source. Each of these computation represents a full evaluation of a neural network. Meaning that we would need millions of training iterations per scene just to compute the lighting of this very scene, and this would be impossible. Each of these neural networks takes a…