Immersive technologies toy with your senses and have the unique ability to catapult you into a new reality. Inside a virtual reality (VR) headset, like Oculus Rift or HTC Vive, we become fully immersed in a computer-generated environment, while augmented and mixed reality (AR and MR) blend real and virtual worlds; they overlay, or mix, our physical surroundings with digital content. Each technology finds a way to turn the seemingly impossible into the possible.
We talked to five immersive technology pioneers working across journalism, filmmaking, storytelling, and scientific research. Their ideas offer a range of perspectives about the potential of immersive technologies. What they all have in common, however, is the belief that immersive technology can be a positive force for the future — depending on what the rest of us choose to do with it.
Nonny de la Peña is an American journalist and co-founder of Emblematic Group, an immersive VR and MR media group. Forbes described her as the “godmother of VR.”
Luis Miguel Samperio is the co-founder of EmpaticaXR, an “evolutionary transformative collective” who are designing the world’s first immersive XR (MR+VR) collaborative and cinematic interactive game.
Mavi Sanchez-Vives is a neuroscientist and co-director of Event Lab, a project that builds experimental VR environments.
Ethan Shaftel directs VR, immersive media, and interactive projects. His VR film EXTRAVAGANZA premiered at the 2017 Tribeca Film Festival.
Carl H. Smith is director of the Learning Technology Research Centre (LTRC) in London. Among other things, he is developing wearable experiences and investigating the ethics of immersive technologies.
It will not just be about becoming other people. It will be about entering the planet’s consciousness.
Medium: Why did you decide to use immersive technologies in your work, and how are you currently using them?
Nonny de la Peña: I want to do what any good journalist does: tell important stories in a way that brings them to life as much as possible, and help the audience find out about, better understand, or feel more strongly about a particular situation.
In Greenland Melting, our second Frontline collaboration, we use the spatial, embodied nature of walk-around VR to let viewers see how far and fast the glaciers have shrunk or how rapidly warm-water currents are eroding the underneath of the ice cap. If we can get people to have a similarly strong emotional reaction to what they see and to get fired up by what scientists are showing them in that piece, then we’re really advancing the medium.
Luis Miguel Samperio: I’ve always realized how difficult it is for people to understand each other and decided that I wanted to dedicate myself to solving this problem using technology, art, and psychology.We are currently encoding different personality patterns and designing ways to represent the inner reality of our minds. We’re doing this through visual and musical representations that users will be able to see and hear when they are immersed in virtual avatars.
Mavi Sanchez-Vives: As a neuroscientist, my first interest was in using immersive virtual reality as a tool to better understand perception. For the last decade, I’ve been interested in virtual “embodiment,” or how we feel a virtual body as our own, what are the mechanisms and the implications of this. I am currently exploring whether virtual embodiment can be useful to modulate pain and also whether it can change behavior—in particular within violent populations.
Ethan Shaftel: Though my background is in cinema, I worked for years creating video content for live spaces — graphics on giant screens for Beyoncé and Rihanna concerts, animations at Disneyland, and 3D pixel displays in a Nike store. In these spaces, you must think about the point of view of the audience in a very different way than in cinema — they participate in the space and in your media. VR struck me as a great medium to combine what I’d learned in immersive design with my love of more cinematic storytelling.
Carl H. Smith: The big project that we’re just about to complete is WEKIT—Wearable Experience for Knowledge Intensive Training. We’ve also got the Seeing I project with [artist] Mark Farid, who’s going to wear the Oculus Rift for 28 days and be somebody else. He’s going to look through their eyes and listen through their ears. He’s chosen 28 days because that’s how long it takes to form habits; he wants to see if he can dissolve his own identity to a certain extent and adopt somebody else’s.
Some technologists believe that because immersion lets us see the world through another person’s eyes, it gives us deeper insight into perspectives that are different from our own. As filmmaker Chris Milk lays out in his TED Talk, this is how virtual reality can create the ultimate empathy machine. Do you think immersive technologies build human empathy?
De la Peña: VR has a unique ability to make you feel present on scene, and that in turn generates a very powerful feeling of empathy. I saw this so clearly with my first piece, Hunger in L.A., about a diabetic man who collapsed and went into a coma while waiting in line at a food bank downtown. When we put audiences through it for the first time, at Sundance, the response was overwhelming: People came out in tears, and even more important, they reacted to the man on the ground as if he were real. They literally got down on their knees to help him.
Samperio: Absolutely. There’s a ton of different examples in the world already, but most of them are based on just embodying someone in a virtual body, putting them in a different environment to become another person, which is fascinating, but we want it to go a step beyond that. We want that to represent the inner reality of a person through their inner voices.
Sanchez-Vives: By experiencing, in a virtual world, the perspectives of others, we can learn to feel how others feel, to be more tolerant and respectful to others. For example, we have given people the experience of changing race or changing age, and the impacts on racial bias and other behavioral responses have been measured; these experiments have had a positive impact on people, enriching their experience and perspective.
Shaftel: I think that empathy is actually a unique weakness of immersive technologies, relative to traditional cinema, and I find it bewildering that it’s discussed as a strength at all. Traditional cinema is an “empathy funnel” that uses sophisticated techniques to take the wildly disparate members of the audience and squeeze them down into the emotional point of view of the protagonist, no matter how different from us they are. We leave ourselves behind and become the protagonist. When we put on a VR headset, however, we change our physical point of view, and the empathy-hijacking techniques of cinema stop working. In VR, we don’t leave ourselves behind, we bring ourselves too, and in some ways are back to the “ground zero” empathy we have in real life.
Smith: All this stuff around empathy could really take off when we have more of the context involved. This question of how you get into somebody’s subjective state is at the core of whether you can create empathic technology.
That line between you and the story dissolves.
How does this technology blur the boundaries between us and the people around us? How far do you think humans will push these boundaries in the future?
De la Peña: It’s the embodied experience. When you feel that bomb go off in Aleppo, you flinch, and you come out feeling shaken. That line between you and the story dissolves.
Samperio: Technology is moving exponentially in the direction of really picking into the inner contents of our minds. There are two researchers who I think are going to be make a huge impact in this area: Marvin Chun from Yale University and Jack Gallant from Berkeley, who are using technology to decode the visuals in peoples’ imagination. Once these technologies are made more accessible and permeate the industry, which is going to end up happening, the phrase “Hey! Would you like to view the things that I’m viewing in my mind?” is going to become real for the first time.
Sanchez-Vives: We are, each one of us, already “many people” in one, in different situations of physical reality. Virtual reality provides a technology that can expand this, because we can experience “life” from the point of view of another person. We are able to experience different human — or nonhuman — beings from a first-person perspective, share spaces with remotely located people, even share a virtual body. Why not?
Shaftel: Immersive technologies blur boundaries between places, which can build empathy just as travel and meeting new people outside our comfort zone can. The promise of VR is “travel” to both real and imagined lands, and a breadth of experience currently impossible to imagine.
Will humans ever be able to essentially become another person using technology, perhaps from a philosophical or neurological point of view?
Samperio: Rumi said, “You are not a drop in the ocean. You are the entire ocean in a drop.” Interpreting this through current technological points of view, I would say that we are all the same entity; we are all the same ocean. We share the same consciousness, but you think you are this little drop and I think I am this little drop. Those drops are going to disintegrate once we embody them through immersive headsets and technologies.
Shaftel: As we’ve seen from social media, interacting through an interface seems to affect our behavior — I might write something in a comment that I would never say to someone’s face — so I think the issue is less about becoming another person, but instead that we become a different version of ourselves.
Smith: It will not just be about becoming other people. It will be about entering the planet’s consciousness, like Moon Ribas from Cyborg Nest does. She has a prosthetic device that shakes every time there’s an earthquake. She’s got a direct relationship with the planet that not many people have yet. It’s not just about wanting to merge with other humans—I think it’s about wanting to merge with other animals, with plants, with the planet.
How do you think humans will use immersive technologies in the future?
De la Peña: Web VR will become the natural way we experience content in the near future. Volumetric storytelling in the browser — no downloads or siloed apps — will make both VR and AR readily created and readily experienced.
Samperio: I’ve just become a dad, and I’ve been thinking about how I would love my daughter to develop empathy, to not feel the sense of disconnection that we all end up feeling once we develop ego. For me, the question is can we create a technological, empathic membrane — through virtual reality — that responds to the needs of our subtle, inner world in the same way as the uterus responds and nurtures?
Don’t get me wrong, my intention is not to put people in virtual worlds so we don’t connect. My intention is to create technology that is very human because it combines psychology and art, through stories, and addresses the inner aspects of our reality, a technology that helps us realize that we are not alone.
Shaftel: These technologies are so young, it’s hard to say. I personally hope to build worlds for people to inhabit that have immediacy and impact and the possibility for catharsis.
Smith: I think the outcome is almost certainly not going to be positive, and that’s why we need to be shouting about it now. That’s why my work is so focused on the ethics of this technology: where it’s going to end up and in whose hands. I’m trying to create narratives around the technology that it’s good for society, that it is going to bring a positive change to your life — instead of the way Facebook wants to use VR, which is to addict you into a virtual space even more than you’ve been addicted into a 2D flat space.