In Ukraine, moving from 360 videos to large-scale volumetric projects
Like many immersive journalists around the world, we at New Cave Media started out making 360 videos. As photojournalists, videographers and graphic designers, we knew how to tell visual stories, and the technical aspects of working with 360 video that seemed overwhelming at the beginning turned out to be manageable.
We shot solitary Arctic landscapes for an interactive application that we continue to work on. We spent time with internally displaced persons (IDPs) who had to leave East Ukraine because of the war, and we made short 360 documentaries about them. Just for fun, this spring we went to a hot air balloon festival and produced a short piece that has captured people’s attention more than anything else we’ve created.
As we were showing our 360 videos to hundreds of people in different cities in Ukraine and elsewhere, I noticed that many viewers were trying to walk while wearing VR headsets. This appeared to be a common phenomenon. People believed what they were seeing, and they wanted to explore the space themselves. I had to explain to them that they were watching a video and that there was no way to really move anywhere.
From the very beginning, we knew that on the other end of the spectrum of immersive storytelling, there were complex CGI-based projects that allowed viewers to physically explore the space. However, as a small team of visual journalists, we didn’t have the resources to create such an experience. So we decided to experiment with photogrammetry.
Photogrammetry is a method of capturing reality that allows you to create 3D models of objects or locations from still photographs. It is a popular method among architects and game developers and was used by journalists before VR really came around (check out this speech by Ben Kreimer at Columbia University in 2014). Now it is receiving increasingly more attention, as it allows you to put the viewers inside a photorealistic world.
An advantage of photogrammetry over 360 video is that it doesn’t require any special hardware. One only needs a DSLR camera (which newsrooms already have at their disposal) or a drone. What it does require is special software like Agisoft PhotoScan or RealityCapture and, preferably, a powerful PC. Compared to CGI you don’t need to build all the 3D models in your experience from scratch; you can “scan” real-world objects and locations using digital cameras (of course, there are limitations). Moreover, you receive documentary 3D models of the real world. In other words, objects and/or locations get “scanned” instead of being recreated.
Our Journalism 360 project, Aftermath VR, grew out of many brainstorming sessions and idea-bouncing evenings that we had as a team. Based on photogrammetry, it is a VR experience giving viewers a chance to walk up Instytutska Street in downtown Kiev, Ukraine, where police forces killed 47 protesters on February 20, 2014, the bloodiest day of the Euromaidan Revolution.
Viewers will learn about and experience what happened that daythrough photographs and video shot there that morning. For us as a team and for me personally, it is a very meaningful project, as most of us were witnesses to the bloodshed that happened that day.
Successfully executing immersive projects like ours is extremely challenging on every level, as so many different worlds have to come together. People of very different backgrounds are working together to make the project happen, and everyone is learning a lot in the process. Researchers gathering data have to understand how dynamic storytelling works to have a better idea of what is expected from them. Photojournalists and videographers have to master photogrammetry and get into the basics of 3D. Directors have to understand Unity (or Unreal Engine) to be able to discuss their decisions with the developers.
At the same time, different backgrounds mean that sometimes there is not much common ground, and often my task as a project lead is to make sure that the team members are speaking the same language.
One of the most challenging aspects of our project is recreating a solid chunk of a street (over a thousand feet) that is located in the downtown of a vibrant city. These are the challenges we face even before the actual shoot begins:
1. You can only do photogrammetry outside on cloudy (but not rainy!) days. This is probably the first time I’ve been happy that Ukraine is so grim in the fall and winter.
2. Downtown gets somewhat crowded during the weekdays, and there are a lot of parked cars, so we try to do most of our work early in the morning on weekends.
3. As the street goes uphill from the main square, there is always something going on, and the scene changes every once in a while. We even had to call the city council and ask them what they were up to in order to have a better idea of what would happen next.
4. The street is in very close proximity to government buildings, so drones constantly lose signal because of the jammers installed in the area.
We will have over 100 different objects in our experience, and each of them has to be rendered in detail. What really helped us was creating a catalog of objects (in other words, a spreadsheet). Structuring the work and micromanaging every piece of the model can have a strong influence on the end result.
Last but not least, photogrammetry is an awesome tool, but just like every tool, it works best when combined with other tools. The list of objects that would have to be 3D modeled from scratch, because photogrammetry could not handle them, grew tremendously from the beginning of the project. Objects that have glass or reflective surfaces or that are thin yet detailed like this lamppost are among the many things that get modeled based on reference photographs.
We are looking forward to sharing in future posts the many things we have discovered about photogrammetry and dynamic storytelling and to sharing the final product of Aftermath VR.
