Volumetric Video: Transforming Visual Media
Ever since the inception of “The Horse in Motion” by photographer Eadweard Muybridge, the field of motion pictures has undergone many revolutionary enhancements. Recorded audio, for example, was first introduced in 1894 with “The Dickson Experimental Sound Film”. Colour in films premiered in 1909 with ‘The World, the Flesh and the Devil’ by James Williamson. And more recently, film digitization has become the industry standard. Each of these developments has added a new level of immersion to the viewing experience and with the recent rise of virtual reality-based media, volumetric video may be the next to do it.
In a nutshell, volumetric video is the viewing of a 3D reconstruction of an object. Traditionally, any sort of movie or video is recorded and observed on a flat screen. With volumetric video, the scenes are recorded as a volume, allowing audiences to experience the scene from essentially any position in space.
The process of recording and developing a volumetric video is through volumetric capture. In order to reconstruct a scene in three dimensions, multiple cameras have to be pointed at the subject from every direction. A volumetric capture studio is essentially just this, a stage that is surrounded by cameras.
The traditional process for recording a scene starts with staging, then filming/directing, and finally post-production. For volumetric capture, these steps differ somewhat: staging, image capture, 3D reconstruction, and then post-production.
Let’s break down the process. In 3D reconstruction, image data from all the cameras are combined to create a single 3D asset. There are a variety of methods to do this; Blue Sky Innovations is working on one which is built using deep learning. For now, the most common methods are by using Light Detection and Ranging (LiDAR) technology. In the post-production phase, textures are placed on the 3D reconstructed model. This process is typically artificial intelligence (AI) based. Post-production processes are extremely crucial because volumetric models are photorealistic, unlike models and environments created in game engines. Whereas a game engine would generate an avatar that may not represent exactly how the person looks, volumetric capture is able to capture the face, body, and clothing of the person.
Photorealism allows for a whole new level of immersion, and many industries have realized this. Members of BTS, a Korean band have used this technology to allow fans to take pictures with their avatars reconstructed in volumetric space. UFC is working with Unity to create a volumetric video of two fighters in order to provide fans with a more captivating experience. Virtual audience members can be the third person in the Octagon, watching punches land, kicks strike, and victors be crowned. In the fashion industry, companies like ANAYI are using volumetric video technology to allow shoppers to view how a piece of clothing looks from any angle, as well as how the fabric moves on the body.
When discussing new dimensions that augment the original motion picture experience (sound, colour, digitization), volumetric capture may be the most significant. It literally adds a new spatial dimension. The audience is no longer experiencing a video from a 2D perspective. They are no longer constrained to viewing camera angles chosen by a director. The storytelling capabilities of this are immense; audience members can view a video from the perspective of a ladybug, or even a bird that’s in the scene. With the rising levels of media consumption through virtual reality, augmented reality and mixed reality, volumetric video may be the next epoch of film history.
