How we developed the project “Lessons of Auschwitz”
The project “Lessons of Auschwitz” is dedicated to the 75 anniversary of the liberation of the Auschwitz concentration camp. Nine Moscow high school students drew in VR their emotions from the visit of the Auschwitz memorial. To make this project we transferred them into their paintings with volumetric video capture technology. The music was written by Peter Theremin, a great grandson of the Leon Theremin, the inventor of the theremin. In the project we combined volumetric/audio reactive Tilt Brush/4K/mixed reality.
Actually school children started to draw their projects in Tilt Brush in October 2019. During that time they visited Moscow Jewish Museum and Tolerance Center and watched the movie “Shindler`s List”. In November, two months before shooting, we went to Poland, where they saw Auschwitz concetration camp for themselves.
For shooting of a 2D version we used one Kinect Azure, DepthKit, Unity, Tilt Brush and Premiere Pro. At first, we wanted to make DepthKit + mixed reality Tilt Brush setup. We gave up this idea because it seemed too risky to shoot nine children and a musician in 4K and mixed reality volumetric in one shooting day that we had. All scenes with drawing children and a composer were captured in a chroma key room, 4K scenes — in the same room, but with the black backdrop.
The idea was to combine somehow painting and music. We invited Peter Theremin, because the theremin seemed to be an up-to-date musical instrument, even through had been invented 100 years ago. While making their VR drawings school children used audio reactive strokes of Tilt Brush, which we combined with the sounds of the theremin to make a kind of a digital orchestra.
During shootings shadows of objects washed their form while moving; some surfaces, usually black, dropped and we needed to stick white stripes on Oculus headsets and some clothes or move the lighting. The idea of the first part was to use volumetric effects for all faces, but then we understood that nine digital faces looked unrecognizable, because Kinect had the limit for detalization and volumetric effects looked similar.
The next idea was to combine 4K and volumetric scenes to imitate digitalization of faces, but then we gave up this too, because it seemed impossible to combine all 4K and Kinect, because of different shooting angles. In some faces they matched well and we left only four with effects.
To combine volumetric children with their drawing process in Unity we made a prototype of future scenes in Tilt Brush. When shooting we asked a child to draw a dot inside his/her Tilt Brush drawing for Kinect camera and then combined their moving with shaders in Unity, which looked like Tilt Brush strokes.
We had a trouble in synchronizing of 4K camera and Kinect while shooting people whole-length, because two cameras couldn`t shoot good from the same distance simultaneously. At first, we wanted to make all drawing scenes in graphic, but then understood that without real children the video looked lifeless. In mixed reality scenes we used screen captured Tilt Brush strokes, which we combined with 4K children's moving in Premiere Pro.
The project was made in collaboration with the studio Phygitalism for the social media project “Pobeda pages”. Site of the project https://en.pobeda.page/vr/
