Responsible Reconstructions: Using Virtual Reality to Expand the Possibilities of Documentary Journalism
The cell measures 15 feet long by 6 feet wide. A cement cot takes up a quarter of the floor space. A fluorescent light overhead casts a harsh glow around the room, and a slit window next to the bed is the only view of the outside world. As you look around, you see a paperback book on a steel shelf, and through a small window in the cell door, a guard in the corridor may catch your eye.
Despite the overwhelming feeing of being inside this solitary confinement cell at the Maine State Prison, of course, you are not. What you are seeing — and feeling — is the first film in the partnership between PBS’ flagship investigative documentary series, FRONTLINE and virtual reality pioneers Emblematic Group. Funded by the John S. and James L. Knight Foundation, the goals of the project are twofold: To produce a series of VR documentaries, and to craft and share best practices for the use of this new medium in the service of journalism.
Our first collaboration on solitary confinement is already pushing at the outer limits of what VR can do. The cell has been captured using photogrammetry, a process whereby high-resolution images are mapped onto a geometrically precise 3D ‘mesh’ of the space. A former inmate has agreed to be interviewed in a lab where an array of more than 40 cameras will film him, moving and talking from every angle. That video recording will then be imported into the cell footage using Unity, a video game platform, to create an experience known variously as room-scale, volumetric, or ‘walk-around’ VR. Users wearing special headsets will be able to move about the space and jointly inhabit the scene with the subject — known in VR parlance as an ‘embodied’ experience.
It’s a heady technological mix that promises an experience like no other, but raises questions, both ethical and journalistic — about how to take advantage of the power of VR while adhering to core editorial principles, and what virtual reality brings to journalism in the first place. We are committed to answering these questions — and pushing each other’s boundaries — as we produce the solitary confinement film and other projects together.
Our partnership kicked off with a symposium at Columbia’s Brown Institute attended by Emblematic, its founder Nonny de la Peña, VR creators from Secret Location, FRONTLINE Executive Producer Raney Aronson-Rath and a group of producers and staff.
Like the solitary confinement project, many of the films we saw that day relied on techniques that we as journalists and filmmakers traditionally approach with caution: chief among them, re-enactments and re-creations. Emblematic regularly uses these techniques in its VR films, such as:
- Use of Force, which recounted the fatal beating of an undocumented immigrant by U.S. border guards. The film was based on two pieces of cellphone footage captured by onlookers. One of those witnesses was inserted into the piece, undergoing body and facial scans so that her movements could be accurately reconstructed; a gaming element allowed users to record 60 seconds’ worth of footage on a simulated smartphone.
- Project Syria was based on found footage of an explosion in Aleppo, supplemented by photographs and maps of the location. An additional refugee camp sequence was modeled on original video footage shot by de la Peña’s team, and employed symbolic figures to represent the number of children affected by the war.
- Kiya was made as a VR companion piece to the Fault Lines documentary Death in Plain Sight. It made use of 911 audio, as well as police and court records, to chart a harrowing episode of domestic violence — two sisters trying in vain to save a third from being shot to death by her ex-boyfriend. The sequence of events was re-created by actors wearing motion capture suits, based on interviews with the survivors; the North Charleston residence was re-created based on crime scene photographs.
FRONTLINE rarely employs re-creations and re-enactments in its long-form documentaries, and when it is used, it’s with caution, guided by long-established standards of journalism. With FRONTLINE’s recent VR films, however, there have been instances where these techniques have been valuable.
Under the FRONTLINE standards, a re-creation can be an effective device for telling a story when the real image or scene is simply not able to be captured — as long as viewers do not mistake it for the real thing. Transparency is essential, as well as visual and spatial accuracy based on reporting, rather than assumptions. If a re-creation doesn’t meet those standards, Aronson-Rath says, FRONTLINE won’t go that route.
With the solitary confinement collaboration, we will be working within these standards as we use cutting-edge VR technology to place viewers inside this meticulously re-created prison cell.
At our VR gathering, several other questions arose about the marriage of virtual reality and journalism:
- What place does the “video game” quality of characters and locations have in journalistic storytelling, and does it undermine the credibility and authority of the work?
- How does the demand for accuracy affect the combining of “real” audio and video assets with material created by the producers? What is the acceptable sourcing for that constructed material — how many sources, and how do you do the vetting?
- What new considerations are needed because of the immersive power of this medium? Could it be “too” immersive and cause harm? Are content warnings necessary?
These are all questions that we have begun exploring together. But perhaps the most important question is the most basic one: Why VR journalism? Why are we doing this?
At FRONTLINE and Emblematic, we may have differing mandates, but when it comes to VR and journalism, we both see transformative potential. Here are some reasons why:
- Presence It’s a cliche, but it’s true. Virtual reality can bring you inside the journalism; it can make your mind believe that your body is somewhere else. In this way, viewers might have a stronger emotional response than they would to something on a 2D screen. There’s also the potential for generating empathy, but that’s something we will approach with care as journalists who endeavor to let viewers come to their own conclusions. Presence is the subject of a significant body of neuro-scientific research, including warnings such as this one about the dangers of a medium that can tap so directly into our consciousness.
2. Spatial Narrative Volumetric VR is uniquely able to convey the physical dimensions of a scene — the distance between a shooter and his victim, for example, or the exact sight lines afforded by a particular vantage point. No other medium comes close in terms of telling a story that hinges on spatio-temporal dynamics — who moved where, and when — especially when it is rooted in rigorous, unimpeachable reporting.
3. Interactivity We are still considering how we will use video game-style controllers in our VR journalism together, but a new generation of headsets allows viewers to become “users” and go beyond being passive observers of a film. These devices let people, for example, pick up and examine a piece of evidence at a crime scene, or switch between the perspectives of different protagonists, deepening their level of engagement with the content by giving them agency within it.
As journalists, filmmakers and VR pioneers, we aim to harness this incredible new technology. We hope to cast light on settings and stories that might otherwise be hidden from view, and, like journalists everywhere who strive to compellingly present what they’ve found, we will try our best to make you feel as though you are there.
Stay tuned.
