The Last Medium
Virtual reality is here. Inside the worlds dreamed up by Hollywood’s most radical storytellers.
By Carina Chocano
Photographs by Holly Andres
A couple of months ago, I met Cory Strassburger, a visual-effects artist and filmmaker, at the creative development studio Kite & Lightning. Strassburger is tall and wiry, with somewhat villainous facial hair on a friendly face. His studio is housed in a fancy branding company in Hollywood and is dominated by a mysterious polygonal metal structure with a platform inside. It looks like a massage table stuck inside a geodesic dome.
Strassburger was excited to preside over my initiation into virtual reality. He placed an Oculus Rift headset on my head and instructed me to adjust the goggles until the Kite & Lightning logo came into focus. Then he told me to close my eyes.
When I open them, I’m in an industrial elevator, going down. (For a panicky instant, I feel as though I’ve been dropped into a nightmare, and I want to get out.) When the elevator stops, I emerge in a tenebrous cavern shot through with shafts of light. Bats scatter in all directions. I look down and see a metal latticework grid suspended over a deep, dark pit. I look up and see a rocky, pitted ceiling high above me. I don’t have a body. It’s like I’ve poked my head into another dimension.
When I took off the headset, I felt as though I’d just removed my head.
I hadn’t thought much about virtual reality since, I don’t know, the last Matrix movie came out. After a month of exploring virtual worlds, my perspective was so radically altered that at one point, milling around the San Francisco airport waiting for a flight, I had a vision of the near future in which all the people around me poking at their phones (which is to say basically all the people around me) had traded in those phones for head-mounted virtual reality displays, and this fad of showing our eyes in public would finally be a thing of the past.
The Oculus Rift — likely on sale next year — has already, almost single-handedly, revived the dormant dream of virtual reality as a mass medium. (Sony and Samsung have similar gear in the works.) Designed by the home-schooled boy genius Palmer Luckey and seed-funded by Kickstarter backers, the Rift combines stereoscopic 3-D, 360-degree visuals, and a 100-degree field of view, which trick the brain into thinking the eyes are looking at actual surroundings. Computer scientists have been messing around with this kind of hardware for more than two decades. What makes the Rift different is that it mostly solves the persistent problem of simulation sickness (if you turn your head and the digital view lags behind, you get dizzy), and it can be made inexpensively. For roughly the price of a video game console, you and everyone you know will soon be able to buy one — which is one reason why, earlier this year, Facebook paid $2 billion to acquire Oculus VR, and CEO Mark Zuckerberg said the company will continue to “invest heavily” in the medium.
Originally, Luckey imagined the Rift as a platform for immersive gaming. But its potential to transform storytelling — through games, yes, but also television and film — could have far wider reach and be stealthily culture-shifting. Stories help us make sense of a world that would otherwise seem chaotic and unpredictable, and derive meaning from lives that might otherwise seem pointless and random. And stories, as Marshall McLuhan famously observed, adapt to the mediums that convey them. The ultimate story of virtual reality may turn out to be a story about stories and how we co-evolve with them.
Filmmakers, animators, visual-effects artists, and journalists have begun experimenting with short features, documentary films, and art projects that immerse viewers deep in other worlds. In February of this year, Oculus hired a director of film and media, Eugene Chung, and started pitching the platform to Hollywood studios. The directors James Cameron, Guillermo del Toro, and Alfonso Cuarón have all expressed interest in the technology. My tour of the virtual reality lab at the University of Southern California School of Cinematic Arts was interrupted — with so much fanfare and advance security that for a minute we thought we were about to meet the president — by an unannounced visit from Will Smith. And this spring, the screenplay guru Franklin Leonard, creator of The Black List, helped Oculus organize a get-together for about 30 screenwriters at the Hollywood Hills home of screenwriter Kelly Marcel (Saving Mr. Banks, Fifty Shades of Grey) to introduce them to the medium and spark a discussion about how it might be used to tell stories.
“I want to see a mystery. I want to see a VR noir,” said one of Leonard’s guests, Joshua Zetumer, who worked on this year’s RoboCop reboot. “Just using the Oculus has a built-in sense of mystery. You strap on the goggles; instantly you’re thrust into an environment that’s not your own. Your first thoughts are, ‘OK, where am I? What am I doing here?’” he said. “You’re like a patient waking from a coma. Or a man washing up on a beach, with no memories of his past life.” Leigh Whannell, who created the Saw franchise, was especially enthusiastic. “Every time I run into him,” Leonard told me, “he says, ‘Man, that was amazing; I can’t stop thinking about it.’” Leonard added: “There was a general consensus that the most obvious opportunity was for horror — but it might be too scary.”
Strassburger showed me another film, a moody, lyrical mini-opera in which I travel down the river Styx from a beautiful mountain landscape into a sinister volcanic hell. At one point, I turn to the left and see the head of a bald, pale giant, his eyebrows and lashes caked with frost. It’s impossible to convey how menacing he is. The ability to get uncomfortably close to someone, a real person or a fictional character, is one of the uncanniest aspects of virtual reality. Last summer, Strassburger told me, he ported to the Rift a photorealistic 3-D model of a bikini-clad Budweiser girl he’d rendered for a demo, so he could look at the work up close. Too close, it turned out. “I had this immediate feeling of, like, ‘I’m invading her space!’” he said. “Being in somebody else’s presence is a huge thing. It becomes a new ingredient for the filmmaker. How does it affect the story? How do you work with it?”
Virtual reality presents all manner of brand-new technical and narrative challenges. Whether or not we’re consciously aware of its existence, we’re all fluent in the language of cinema. We’re native speakers. Virtual reality is in such a germinal phase that to use it as a medium to tell stories is to participate in creating a whole new language. The morphemes of cinema — framing, cutting, close-ups, pans, zooms — disappear or stop making sense. It’s no longer obvious how a filmmaker directs attention or advances plot. What about perspective and point of view? Should the viewer be free to explore a world or follow a set path through a story — on rails, as gamers say? Should characters respond to your presence?
Finally, it was time for me to climb into the bed-dome contraption, affixed with a dozen speakers. I put the Oculus headset back on. The view through the headset looks just like the room around me. At first I think I’m looking through the goggles at the ceiling. Then, music begins to vibrate through my body, and the ceiling detaches and flies away, and I’m floating in space. Small fiery suns swirl around me and hover so close I have to stop myself from reaching out. As I soar around this vast, luminous, imaginary cosmos, I surrender completely to the experience.
In her magical book, River of Shadows: Eadweard Muybridge and the Technological Wild West, the author Rebecca Solnit pinpoints an origin of the modern world — or more precisely, the “amalgamation of technology, entertainment, and what gets called lifestyle” that we now recognize as everyday life in much of the world — to a pair of eminent Californians and some pictures of a horse.
The photographer was the artist, scientist, and self-invented inventor Eadweard Muybridge, whose high-speed photographs of a fast animal in motion laid the late 19th-century groundwork for the invention of the motion picture. The owner of the horse was Leland Stanford, one of the “big four” industrialists responsible for the transcontinental railroad (also, Stanford University), who commissioned Muybridge’s motion study in order to settle a debate. If Muybridge’s invention seemed to capture and manipulate time, Stanford’s collapsed space, shrinking the distance between the coasts and creating the conditions for the standardization of everything in between. Two industries would eventually sprout from their work, and both would come to be known — ironically, given their inherent timelessness and placelessness — by their geography: Hollywood and Silicon Valley.
“Annihilating time and space,” Solnit observes, is what many of our most transformative technologies have aspired to do. Stories — and storytelling mediums — aspire to do the same thing. Like trains and telephones and everything that came later, they’re designed to transport us in time and across infinite distance.
Two decades after Muybridge photographed Stanford’s horse, a pair of French brothers, Louis and Auguste Lumière, publicly screened their film, The Arrival of a Train at La Ciotat Station. There’s a well-known story about how the audience panicked and fled from the oncoming train. The story is apocryphal, it turns out. Still, even if the Lumières’ audience members knew they weren’t actually in mortal peril, they were probably still freaked out by the discovery that the eyes and brain can conspire to trick the mind. Not surprisingly, the train story came up often as I spoke to filmmakers contemplating a new kind of illusion. Eugene Chung first mentioned it as a way of explaining the epochal media moment we find ourselves in — much like the late 1800s all over again.
“It’s bigger than color,” said Chris Milk, the influential music-video director, multimedia artist, and technologist. “It’s bigger than sound. It’s the audience literally inhabiting the narrative.”
“The fact that we now have a presence within the story makes a profound difference on a biological and emotional level,” he said. “They’ve done studies that show that our memories are geotagged. Memories are embedded with location.” Professor Mark Bolas, director of the Mixed Reality Lab at USC, agrees. Bolas began creating rudimentary virtual worlds 20 years ago and recalls them as fondly as a European backpacking trip he took. “I’m nostalgic for worlds. To me, those were very real places. They don’t exist anymore because the computer’s gone, that head-mount’s gone,” he said. “I’m not the only researcher to say that. I’ve talked to some other people from the old days, and they’re, like, ‘Yeah, I miss my worlds.’”
Last year, Milk filmed the artist Beck and his father, composer David Campbell, performing an elaborate arrangement of David Bowie’s “Sound and Vision” — in the round, accompanied by more than 200 musicians, including an orchestra, a gospel choir, and a gamelan ensemble — and later stitched the footage together to put a virtual reality viewer in the room with Beck. “People have watched this and talked to me months later and said that the way that they remember it is not as a piece of media that they consumed. They remember it as a place in time that they existed within,” he said. “It’s making a deeper memory.”
Virtual reality is at once a communication platform and a transportation medium. It completely annihilates time and space.
I’m standing in an elevator again, this time going up. Had I experienced this Rift film as it was originally experienced, by ecstatic hordes at this year’s South by Southwest festival, I’d feel the thunder of a rumble pack beneath my feet and, upon reaching the top, a sudden gust from a wind machine on my back. Then I’d turn to face the icy blast and find myself high on the Wall, from Game of Thrones, staring down a long corridor. I float down the corridor to a vertiginous ledge, overlooking the austere frozen valley beyond the Wall. At first, I’m mesmerized by snow flurries. Then I notice a marauding army coming into view. A flaming arrow shoots through the air, then another, then a close one. The fourth one hits its mark, and my arm is on fire.
Seriously, my arm is on fire. I flap it around like a doomed turkey.
I was in Framestore’s Culver City office. Framestore, a visual-effects studio that won an Academy Award for its work on Gravity, was an early backer of the Rift on Kickstarter. Mike Woods, the executive creative director, said they were commissioned by a live-events company to create a Game of Thrones experience that would take festival-goers up the Wall. They knew how to create a virtual environment. The first 13 minutes of Gravity is just that, one ostensibly continuous shot that Framestore stitched together from traditional footage. Turning the audience into protagonists, directing the viewer’s body, this was new.
Woods wanted his audience standing up. The idea faced resistance from Oculus — they thought the audience should be seated — but Woods said the company loved being proven wrong. Should you, the viewer, have a body? At first, they were convinced you should, and it was fine as long as you stood still, but being scooted forward when your actual legs weren’t moving felt horrible. Perhaps not surprisingly, a disembodied consciousness is easier to scoot around. “We learned the hard way what you can and can’t do,” Woods said.
As someone who went to film school and studied mise-en-scène, classical narrative, and framing, Woods finds it intriguing that he now has to learn a whole new language: “You can’t cut. You can’t fade. You can’t move the camera. You can’t pull focus. What you can do is create an intriguing hybrid of Brechtian in-the-round theatrical stuff and game design.” If you’re going to get hit in the chest with a flaming arrow, before that can happen, three perfectly timed and directed arrows have to be released (one to call your attention, one to focus it, and one for good measure) so that by the time the fourth one hits, you’re looking right at it.
For several years, Félix Lajeunesse and Paul Raphaël, partners in the Montreal-based Felix & Paul, have explored different ways to create cinematic experiences without relying on traditional storytelling conventions. Recently, they developed their own technology to capture live-action virtual reality. The camera, which is under wraps, so I don’t get to see it, not even a picture, has multiple sensors and looks — I’m told — kind of like a person, with eyes, ears, and a body underneath. Lajeunesse told me to imagine a small humanoid robot, which I did, and it’s cute.
Lajeunesse and Raphaël are big on “experiences of presence,” which is to say, creating films that foster a sense of connection to other people, a place, and a moment. They reached out to the Canadian singer-songwriter Patrick Watson and asked to record him in his studio. “We basically said to him, ‘This is not a camera; this is a person.’” The camera was positioned where Watson indicated a guest in his studio might sit, and then the filmmakers left the room.
“Cinema tends to be really preoccupied with always moving on to the next thing, cutting to something else,” Lajeunesse said. “But in VR, we feel an urge to slow down, to land and really explore the moment.” I understand what he’s saying. We live in a time when we are constantly distracted, constantly privileging the ephemeral over the real. Lajeunesse and Raphaël want viewers to stay focused on a moment for five, six, even ten minutes. “For us, it’s not just about the content itself,” Lajeunesse told me. “It’s also about creating the right conditions for the viewer to feel a part of it.”
What happens in Strangers — A Moment With Patrick Watson is pretty much nothing, and yet it might rank as one of most profound aesthetic experiences of my life. There’s something about how intimate, still, humble, small, and graceful it is that’s deeply moving. Watson sits at his piano, noodling around. He smokes a cigarette, or mostly lets it burn in the ashtray. Smoke rises and curls. His phone dings; he looks at it and laughs, puts it down again. Once or twice, he looks at the camera as if acknowledging my presence.
Lajeunesse believes that trying to make deliberately “exciting” or “interesting” experiences will only make the viewer feel like he’s consuming media, in which case “the immersion will have failed.” But if you create a scene that a viewer can inhabit the way we inhabit time and space, then his imagination and sensitivity and mood will come into play, and he will gradually start to feel like he’s there. “His sense of anticipation changes,” Lajeunesse said. “His desire to move on to the next thing is reduced. He starts focusing on what is there, at that moment, in that instant, place, and time, on the human being.”
Before switching from film to virtual reality, Lajeunesse documented Inuit societies in the Canadian Arctic. Lately, he has thought about returning. He’s interested in “going around the world and capturing moments of reality, whether it’s a family meal or a prayer or a ritual or just kids playing around,” he said. “Trying to archive those moments of reality for now but also for the future.” It’s a beautiful dream, the idea that virtual reality could evolve into a kind of existential lepidoptery, dedicated to the melancholic and somewhat perverse pursuit of capturing and preserving the evanescent moment, the bygone place, the traditional way of life before it is transformed by technology.
Everybody talks about empathy. Virtual reality will allow us to feel compassion for people worlds away. “It’s this chance, I think, to redefine how we live in this world and understand ourselves,” said Danfung Dennis, a former war photographer and documentary filmmaker now working in virtual reality. “We’ve been limited to this one viewpoint; you’re born and you die looking out these eyes. Then all of a sudden, we’re able to slip on someone else’s consciousness.”
I’m not so sure. Appealing as it is to imagine virtual reality as a medium that puts us in other people’s shoes, that’s not the same as putting us inside their heads. Film is symbolic. We project ourselves onto characters on a screen. We know they are not us, but we watch them to discover the ways in which they are like us. The psychoanalytic film theorist Christian Metz describes film as a mirror that does not reflect the spectator. Instead, it forces the spectator to identify with something else — the camera. The movie spectator is “absent from the screen but certainly present,” Metz writes, “a great eye and ear without which the perceived would have no one to perceive it.”
Perhaps one of the negative effects of technology on art is that it encourages a conflation between verisimilitude and “reality.” Does “entering” a virtual refugee camp as someone (you, me) who is not hungry or ill or in any way traumatized come close to replicating the experience of actually living in one?
Part of the allure of annihilating time and space is our desire to eradicate the unknown, to make everything visible, to reduce our primitive, mystical reliance on our imaginations. Yet empathy depends on imagination. A Virginia Woolf novel may not actually transport you to the streets of London in the early 20th century, but it plunges you deep into her characters’ subjectivity. I’m reminded of something I read on the author Alain de Botton’s School of Life blog: “A novel is a machine for simulating experience.” A “life simulator” that allows us safely to experience what we otherwise couldn’t. “Unaided, we are puny in our powers of empathy and comprehension, isolated from the inner lives of others, limited in our experiences, short of time, and able to encounter only a tiny portion of the world firsthand.”
The birth of a new medium inevitably yields a rush of inventions — once people saw Muybridge’s new camera, the zoopraxiscope, soon there was a kinetoscope and a nickelodeon. Similarly, the release of Oculus Rift prototypes has inspired ambitious artists to hack together new systems for capturing and recording immersive video and audio.
Most filmmakers, of course, just want to make films. Jaunt VR, which is based in Palo Alto and is located somewhat ingloriously above a mattress showroom, is developing a system for capturing and delivering virtual reality films: cameras, software, and, eventually, a player. The version of the Jaunt VR camera I saw — an orb covered with lenses, each pointed in a different direction — has been described as a disco ball, and there’s definitely a glancing resemblance. But what really distinguishes it is the math. Every vantage point is captured by multiple lenses, and then the camera computes the data, and turns it into a spherical image for each eye. The point is, the image is not so much captured as computationally assembled. It’s not a record; it’s a construct. We are talking not about things here but ideas. “It’s a notional camera,” said Scott Broock, Jaunt’s V.P. of content. As for what it creates: “You can think of it as image, but really it’s just data for the brain.”
In a small, dark room at the back of the office, Broock showed me a series of live-action virtual reality clips that were some of the most immersive I’ve seen. Shot at 60 frames a second — more frames and therefore much sharper than traditional film — the footage is startlingly detailed and life-like. “A lot of people say, ‘Well, maybe if you shot at 24 [frames a second] it would look better,’” Broock said. In other words, more like a movie. “But that’s the whole point. This isn’t film. It’s something completely different. It’s all about experience.”
Broock put me in a headset, and the next thing I know I’m on the ramp at a skate park, then in the front row at a boxing match (for a client that I can’t identify), then onstage at an electronic-dance-music concert (again, top secret). Then he showed me a scene from a sci-fi action movie. Suddenly, I’m on the deck of an Arctic research vessel where the crew is being torn apart by alien monsters. Wherever I turn, someone is getting impaled or dragged away.
I told him I was amazed. I really was. “You’re a writer,” he said. “If you could tell me how you could accurately and convincingly express what that thing is, that experience to someone who’s never tried it, I’d love to know the words, because I still can’t figure it out.”
OK, how about this: The kid at the skate park zooming past me and flying overhead makes me feel invulnerable. I’m there, but I know I won’t get hurt. At the match, I’m so close to the athletes — or rather, I’m so aware of my place near the action, surrounded by the keyed-up crowd — that my body leans forward and my muscles tense with communal excitement. Confetti explodes overhead at the concert. It feels like flying through a cloud. Watching the fictional massacre on the ship, I feel like I’ve witnessed a private snuff show.
It occurs to me that not only am I getting used to this disembodied-consciousness business, I’m actually quite enjoying it. I also notice that I’ve slowed down on asking myself the big, open-ended, existential questions that plagued me the first few times I entered a virtual world: Why am I here? Am I really me? I truly feel transported and immersed, but I’ve stopped wondering about where or how I fit in. On the contrary, I feel invisible, like an all-seeing eyeball. I feel entitled. Like the world exists just for me.
I think of something Chris Milk told me. “This is the last medium,” he said. “We’re at the very beginning of it, but version 147 is The Matrix or Total Recall. Our brain is no longer translating an approximation of the story. You read a book; your brain reads letters printed in ink on paper and transforms that into a world. You watch a movie; you’re seeing imagery inside of a rectangle while you’re sitting inside a room, and your brain translates that into a world. And you connect to this even though you know it’s not real, but because you’re in the habit of suspending disbelief.
“With virtual reality, you’re essentially hacking the visual-audio system of your brain and feeding it a set of stimuli that’s close enough to the stimuli it expects that it sees it as truth,” he said. “Instead of suspending your disbelief, you actually have to remind yourself not to believe.” Our brains, after all, are like notional cameras. We construct our reality out of data.
Carina Chocano is a contributor to The New York Times Magazine. Her book You Play the Girl will be published next year by Atavist Books.
Holly Andres is a photographer based in Portland, Oregon. Next spring, she will be exhibiting at the National Museum of Photography in Bogotá, Colombia.
Originally published at californiasunday.com on October 5, 2014.
