Let’s Get a Grip on Virtual Reality
From ACM Interactions / Nov — Dec 2016
How do you interrupt someone plugged into a virtual reality (VR) system?
I was faced with this peculiar issue when meeting with a colleague working on VR. Ambling over to his desk amid an open-air setup of cubes and pods, I noticed right away that he was fully plugged in. Giant dark goggles covered his face, his head bobbing and weaving while his hands gripped two controllers, making jerky movements as if he were playing air guitar. Yup, he was in another world for sure! So how could I approach him and let him know I was physically there for our (now twice rescheduled) coffee chat? How to ease him back into this world without jolting him, causing him to mess up whatever virtual work he was doing? Perhaps there’s an as yet undiscovered graceful mannerism I could use …
VR is creating some curious conundrums that need solutions — but hey, this is totally apropos for experts in all matters human-computer interaction, right? It’s a rich problem space, no doubt. Especially as VR is now all the rage. There’s excitement over Oculus, a Kickstarter project acquired by Facebook that ships as the Oculus Rift and also Oculus Gear via Samsung; HoloLens, a Microsoft headset that enables interacting with holographic imagery; and startups like Magic Leap — based in Fort Lauderdale, it promises photorealistic projections that interact with your physical surroundings, which is called MR, or mixed reality. And let’s not forget AR (augmented reality), as epitomized by the runaway success of Pokémon Go, serving up a layer of visual data on your phone and transforming it into a digital divining rod to find virtual creatures around your actual neighborhood.
Indeed, it appears we have reached a technological tipping point that has made VR truly plausible and usable, a far cry from the blocky polygonal graphics of the 1980s and 1990s, which demanded tremendous resources yet delivered meager and frankly laughable results. Lawnmower Man, anyone?
But as VR and its variants seem to be expanding in pop culture, I can’t help but feel déjà vu over this notion of being sucked into an alternate realm that affects your mind, body, and emotions.
Think of the following situations. They all feel quite similar, no?
- A bus rider deeply absorbed in their mobile phone, maybe a social media feed like Facebook or a funny cat video on YouTube, misses their stop (again).
- Binge-watching TV shows on Netflix or Hulu, losing track of time, being pulled along by the mesmerizing plot lines and dramatic cliffhangers, while popcorn scorches on the stove.
- Blissfully reading, engrossed in a book, off in your internal world of mental imagery painted by poetic words, just barely noticing the doorbell of the delivery guy dropping off a package.
- And a quick Throwback Thursday reference: Kids playing Dungeons & Dragons (D&D) in the 1980s, collaboratively shaping a fantasy world of danger, heroism, and riches, getting lost for hours in the basement while parents upstairs yell to help with chores.
In each of these examples, ordinary folks are ensnared into a kind of reality via some medium that enables multi-sensory engagement at physical, cognitive, and even emotional or spiritual levels. These scenarios mix intangible value with intuitive and attractive qualities that foster a kind of temporary residency or habitat for sustained periods of time, but with the loss of perception of time. They’re truly autotelic experiences that are either self-enhancing (for individuals) or communally empowering (for groups).
The current VR revolution intensifies those qualities, thanks to high-resolution imagery with exceptional degrees of color, depth, and texture — qualities that enable a sense of entering a visceral realm complete with spatialized visual and auditory cues, even to the point of causing nausea or disorientation. It’s a fully 360-degree, seamless, continuous space, perhaps infinitely extensible, much like Neal Stephenson’s Metaverse or William Gibson’s Cyberspace. Indeed, how can one tell the difference between the real world and the dream world?
Whether it’s the virtuosity of an emotionally rich book or the virtuality of a digital palace that occupies your entire field of vision, each embodies a fascinating set of issues that HCI experts are effectively skilled at dissecting, if not fully resolving. It’s high time we tackle these challenges, before VR proliferates out of control!
First, let’s acknowledge that designing for VR demands a rare blend of skills: game design, 3D modeling, film/theater choreography, architecture and interior design, urban planning — and yes, even typography still matters! Like, how does one interact with dimensionalized type that advances and recedes, or that one can virtually walk around like a sculpture in a museum, seeing the shape, shadow, and light warp in real time? What would reading email or a social feed look like transposed onto VR surfaces? How does one “click” an ad in VR? (Hey, we know that’s coming!) And how does one navigate VR space effectively to interact with (or ignore) sensory cues begging for attention? What is that kind of workflow like? Our expertise in defining online task flows and page structures brings considerable value here.
But that’s at the level of what I call the interior dynamics of VR — the interactions and behaviors of objects in virtual space as digital representations seen by the goggled-up viewer, similar to buttons and icons on a Web application screen.
Going deeper into such VR capabilities, what about the issues of reality and social engagement, which raise a range of troubling questions from a humanistic perspective? We may conceive of VR as a “consensual hallucination” (per Gibson) that’s seemingly infinite — but is it truly? Does it make better sense to expose the seams between virtual and physical spaces, or levels and zones within virtual worlds? How do the sensations of the physical world (sounds, bumps, and textures) make their presence felt inside a VR context — and should they? Should we allow for a VR participant to somehow pull back the curtain to reveal the artifice driving the simulation, especially after extended periods of time? Hmm.
I remember as a kid playing video games for 4 to 6 hours at a stretch, being completely exhausted and zoned out. What’s that like with something that fully occludes your core senses from the real world? Do we introduce a “totem,” like in the movie Inception, to help someone maintain their orientation to reality while within a labyrinthine virtual space? It’s no longer just about the discoverability of features, but rather the findability of yourself. Where are you truly? Do we prompt for a “kick” to push the user back into reality after some time — or depth — limits? What are the mechanisms and ergonomics to make all that happen?
There’s undoubtedly a heady mix of profound issues at the heart of VR that cannot be simply “tech’ed” away — presence, reality, identity, levels of interactivity with impact, virtual boundaries and gateways. The outsider’s POV of the VR participant, which entails its own silent discourse of observation and knowing when or how to engage. The insider’s POV of the VR space, with its own interior dialogue of interactions and affects. Plus, naturally, how is all this being tracked, captured, shared, and logged by corporations and governments alike (again, we know this is all coming). And, of course, for the VR participant — or shall we say actor? — what are the range and limits of actions that can or should be taken? How close can they, or should they, mimic the real world? For instance, we decry offensive bullying comments by trolls and such on Twitter and Facebook — but what if you can physically intimidate and beat up someone’s avatar in a virtual realm? What are the legal and ethical scopes of affecting virtual human behavior? How can we, as advocates for a humanized expression of technology, impart values of propriety and safety in defining virtual contexts through a human-centric practice, without appearing like Big Brother? Tough questions!
It may be useful to examine this brave new world of virtual ambiguity through a simple framework — intention-consequence-threshold — as a way to help evolve the affordances and allowances within a VR context:
- Enabling the intentions of VR actors as cognitively originated by the real human being and virtually expressed via their avatars’ actions, words, and emotions (i.e., simulated representations).
- Mapping those intentions to consequences within that space, which itself is affected by the real world within which the virtual is being consumed or projected. This goes beyond simple interactivity and feedback loops; it is actually the consequence of taking certain actions with virtual objects, projected avatars, and related expressions (i.e., floating emojis, virtual ads, or promoted elements). With gestural movements, you could duck, leap, kick, slap, punch, grab, maybe even bite — the whole range of actions we do in the real world, presumably! Thus, what are the kinds and levels of consequences we are willing to allow and what are their ripple effects into the real world? This demands some serious understanding and discussion with the stakeholders and communities the VR systems serve. You can’t just file a JIRA ticket or debug the code before launch.
- What are the thresholds for acceptable/tolerable actions in virtual space? What are the criteria for deciding how they are developed, shared, and then subsequently enforced? This is where defining (collaboratively, from a strongly HCI-oriented posture, of course) the algorithms and interactive elements for a virtual space becomes vital. We cannot leave this up to programmers! For example, it simply might be the equivalent of flagging a bullying comment in the Facebook feed or citing an ad as being offensive. Given the richness of VR worlds, what can we do beyond that? A critical blend of thought and creativity is needed.
Indeed, VR and its variants dazzle our senses with exciting possibilities, yet they embody profound challenges at a personal level. How we tackle such emerging technologies with their social and humanistic implications as practitioners of HCI will define who we are as a field. How can we truly shape a future that supports and improves the human condition? Undoubtedly we have a variety of tools and lenses to help us tease out deeper issues and propose intelligent solutions, or at least pathways for thinking through them. It’s worthwhile to apply HCI-based tools to understanding such issues and also to pull things into a, well, virtual balance benefiting everyone.
Copyright held by author. The Digital Library is published by the Association for Computing Machinery. Copyright © 2016 ACM, Inc.
Originally published at https://interactions.acm.org.
