Why AR? How we leveraged the strengths of 3D to produce a mobile narrative on a complex subject
By Cassandra Herrman and Nani Sahra Walker
Cassandra Herrman and Nani Sahra Walker are the producers of What They Carried, which tells the story of migrants crossing the Mexico-U.S. border through the personal belongings they choose to carry with them. They were part of the first McClatchy New Ventures Lab Storytellers in Residence cohort. Download Actual Reality for Android to experience the augmented reality series.
“What are you working on now?” As independent filmmakers and journalists, we’re asked that question constantly. Over the past several months, answering has been akin to a game of charades, mixing pop culture references with pantomime. “Do you know Pokémon GO? It’s like that plus journalism.”
Sort of.
As part of the inaugural cohort of Storytellers in Residence at McClatchy New Ventures Lab, we were tasked with creating a journalistic augmented reality series for mobile. Which means we’re trying to get beyond Pokemon-plus or Journalism-lite. That’s a tall order in a medium that is still finding its sea legs. At the moment, AR is predominantly visible in Snapchat filters and utility apps, with a handful of news organizations like The New York Times, USA Today and The Washington Post publishing AR nonfiction content.
If you believe Apple’s CEO Tim Cook, augmented reality (AR) is going to “change the way we use technology forever.” AR mixes the digital world with the real world in real time by superimposing graphics, data and other information via the built-in camera. Unlike virtual reality, or VR, which immerses you in a virtual world through a headset, AR experiences appear right in front of you, wherever you are.
AR was appealing to us because it’s a nascent medium that’s ripe for experimentation and like VR, it’s a new way to engage audiences. The platform also promises greater distribution potential than VR because it will be supported on all mobile phones in the near future.
With backgrounds in documentary and journalism, we wanted our series to tackle a topical issue, yet we also knew that we would face a common VR/AR roadblock: context and narrative information don’t always play well with immersion and virtual overlays. This becomes even more glaring when you’re designing for an experience on a mobile phone.
Which begs the question: Why AR?
Why tell a journalistic feature story in a medium that seems designed to showcase bite-sized content? What could AR offer us as storytellers that print, video, photography or audio couldn’t?
One word: 3D.
One of AR’s unique strengths is that it allows you to place, interact with, and explore three dimensions in your own environment. Playing with scale, perspective and interaction changes the way we experience stories. Imagine seeing the first guitar that a famous player used in front of you, and being able to hear the strumming sound it really makes, or looking down at a tiny diorama of a burned down neighborhood and understanding the collective loss by seeing — at once, up close — every house that was lost.
We started to brainstorm journalistic stories that had real-world objects as their centerpiece. And that’s when we came across Tom Kiefer. Tom is a photographer who moved from Los Angeles to Ajo, Arizona, a small town 40 miles from the border with Mexico in 2001. To make some extra money, Tom took a part-time job as a janitor at the nearby U.S. Customs and Border Protection processing center. One day, while going to retrieve discarded food items to donate to a local pantry, he noticed that amongst the trash were bags full of belongings that had been confiscated from undocumented migrants who were apprehended trying to cross into the United States. He said officers were told to seize any items that are considered “non-essential” or “lethal.”
Tom started to pay closer attention to the items, which when viewed en masse told a deeply revealing story of the migrants’ journeys: children’s toys, flashlights, soap, keys, CDs, love notes, religious candles. He began saving them, and eventually started a photographic project documenting the objects titled “El Sueño Americano” (The American Dream).
We contacted Tom and explained what we hoped to do and he was open to us coming to Ajo to capture some of the items. Tom refers to himself as a “custodian” of the migrants’ possessions, and only asks that journalists treat them with respect and dignity.
Overlooking the main plaza in Ajo, Tom’s photography studio is filled with thousands of items collected over the course of seven years. We knew we could only include a dozen or so in our AR experience so we chose ones that would help us advance the narrative and illuminate important story themes.
A water bottle, flashlight, backpack and insulin box convey the logistical and safety challenges of crossing the desert; a child’s toy, quinceanera napkin and favorite music CDs illustrate the emotional ties to home.
Using a process called photogrammetry where you take hundreds of pictures from different angles using a basic DSLR camera, and a software called Reality Capture, we were able to produce a high resolution, 3D model of each object.
We knew we also wanted to convey a sense of place, as well as give the user some kind of connection to a real person, whether that was through a video interview or audio voiceover. To capture locations and interiors, we used 360 video and drone photogrammetry, and for subjects we experimented with a volumetric video software called Depthkit. Depthkit is one of the more accessible volumetric tools on the market, and works with an affordable depth sensor (the Microsoft Kinect), an HD video camera, and a computer.
Finally, we loved the idea of using 3D illustrations and animations to enhance and contextualize the photo-real objects, create background environments like the desert, and give our text captions a handwritten, creative feel. We also used 3D illustrations when we couldn’t get a 3D reconstruction, either due to access or privacy considerations.
To achieve this, we used two VR painting and animation tools, Google’s Tilt Brush and Oculus Quill, and worked with Micah 404, a talented designer and artist based in New York.
All of these elements were brought into Unity, a game engine platform, where our developer Julian Rojas assembled and scripted the diverse parts into a coherent narrative.
Once we had the technical tools and pipeline down, we still needed to figure out how to create a cohesive, contextual, journalistic series that had some kind of narrative arc. We settled on three episodes using two different styles: tabletop and room-scale. Our first episode, “Belongings,” is a tabletop experience that introduces confiscated items through photographer Tom Kiefer.
Episode two, “Crossing,” flashes back to the precarious journey across the Sonoran Desert that so many migrants make, including the Dreamer we interview in episode three — Karina Ruiz. Micah created five distinct desert environments in Quill, representing the passage of time from day to night. He animated elements in the room-scale scenes: a flashlight beam, the silent flight of an owl, and a thermometer that shows temperatures shooting up to 120 degrees.
The episode pulls together objects of unknown origin; we don’t know who owned the objects and what their individual journeys were like. The illustration aims to recreate the feeling of the journey, referencing images of the southern Arizona desert to represent a collective experience, but doesn’t depict a specific place.
We wanted to add more context and narrative stakes to the experience but didn’t want to overwhelm the user so we incorporated 2D photos with captions and experimented with data visualization. We featured photos from the Colibri Center for Human Rights’ “Bring Them Back” campaign, which helps families searching for loved ones who have disappeared along the Mexico-U.S. border. Using family photographs combined with text, we were able to integrate personal stories into our desert scene. For the end of the experience, we took data from the Pima County Medical Examiners, Humane Borders and U.S. Border Patrol of how many people have died in the Southwest desert over the past two decades to create a visualization in the night sky using particle effects.
Our final episode, “Dreams,” features Karina Ruiz who made the journey from Mexico across the desert with her parents in 1999 when she was 15 years old. A Dreamer, who now runs an organization helping other migrants, Karina carried a few precious belongings with her from home. Since then, other belongings in her life have been imbued with meaning, including the college graduation stole that took her 12 years to earn.
Through Karina’s memories, and her hopes for the future, we bring the series narrative full circle. In this tabletop experience, we revisit many of the techniques in our first episode, from photogrammetry to Quill, including a centerpiece salsa dance that Micah animated to bring Karina’s quinceañera celebration to life. For her Depthkit interview, we worked with the talented Andrew Gant and New Venture Lab’s Jayson Chesler, stylizing the volumetric video to achieve a look that mixes photo-real and digital. We chose this direction because we felt a more standard video look in the context of the scene’s animated aesthetic could be distracting and pull them out of the experience.
As we neared the end of our 6-month residency and prepared to launch the series, we took a moment to ask ourselves, again, Why AR?
Did we feel any closer to answering that question?
Yes. Sort of.
Even with all the strangeness of this new medium, our hope is that what it does offer to journalism — the chance to see and experience things up close and personal, wherever you are — will offer a new angle on issues that deserve one. The same way that we felt deeply impacted standing in Tom Kiefer’s studio, surrounded by thousands of belongings that threaded back to individual lives, AR can bring some of those objects, and stories, into homes and offices and into our daily lives.
As journalists and immersive storytellers that possibility is worth the risk, and the challenge of experimenting, to try and get there.
For us, at this moment, it’s Why enough.
