Lessons of Auschwitz: VR-project created with the help of Volumetric capture. Part 1

Published in

PHYGITAL

4 min readAug 6, 2020

On January, 27 we posted a video art project that we had been working on for 2,5 months. The series of articles will be devoted to the concept of our project, both meaning and technologies aspects. We will also show you why you should not fear the unknown.

Part 1 \ Part 2 \ Part 3 \ Part 4

In 2019 many winning projects of international competitions and festivals were created with the help of volumetric capture (video technology). Dimension Technology, who worked on Madonna’s live performance at the BBMA’s, became a winner of Broadcast Awards. The HALO Awards at VRDays in Amsterdam also recognised such works, and VR Church used it in a number of projects. Volumetric video projects were featured in Kaleidoscope funding platform, on immersive exhibitions, technologies conferences and festivals all around the world. The music video production industry has also been using this technology. One of the first music videos done with Volumetric capture was Radiohead’s House of Cards from 2008. We felt that we should accept the challenge and jump at the opportunity to develop our professional skills.

Fragment from the site of the VR-project “Lessons of Auschwitz”

As for the meaning of our project and the idea that stands for it, it is dedicated to the 75th Anniversary of Auschwitz liberation. Children who drew pictures for our project are representatives of the generation who did not get to meet participants of those events. Children went to Poland, where they listened to lectures and watched films in order to get a full picture of events. They expressed their emotions through their VR-projects, which became the prism through which the current generation sees the past.

The music for the project was composed by Peter Theremin (his grandfather, Leon Theremin, is famous for the invention of theremin). The final version of our project combined such technologies as volumetric capture, audio reactive brushes from Tilt Brush, computer graphics, and mixed reality. Denis Semyonov (SaintDenis), a famous VR artist, became the producer of the project and he managed the children’s training.

Art project “Lessons of Auschwitz”

The task of PHYGITALISM team was of a technical character: we conducted a research on the mechanics and technologies that would work best for the project, recorded the participants with depth cameras, developed visual effects using Unity Visual Effects Graph, processed VR images and imitated painting in mixed reality. At the final stage of our project we made it possible for the children to appear in their own pictures with the help of volumetric capture technology (Volumetric).

Fragment of the video art “Lessons of Auschwitz”

Computer graphics was of paramount importance, we have selected the visual effects according to the project’s mood. All these effects were chosen and adjusted to suit the participants’ personalities and mood. As a result, we managed to compile a collection of different Volumetric interpretations.

Effects used in the project

Such projects as this one give you an opportunity to work in a chroma key studio (a room with a green screen) and get an invaluable experience. Not only did we film in 2D there, but we also achieved a more high-quality motion capture with help of DepthKit library and Azure Kinect sensors. All of it enabled us to achieve the effect of Volumetric Capture. DepthKit has been widely used in projects that were awarded with prizes, for example, Vestige video art.

We also would like to mention that James George, the creator of DepthKit, gave our project recognition.

Backstage from filming

In the next articles we will provide you with a better understanding of the technologies used, explain some subtle aspects of the project, and there will be much more terms. We would like to share the pipeline of our project with you. You will learn about the research we had conducted, about the process of the production with the help of Unity 3D, DepthKit, After Effects. We will also tell you about the difficulties that may arise while filming with Kinect Azure, how images can be captured from Tilt Brush, and, of course, we will tell you about the production of computer graphics in the project.