Designing for Hands in VR

The next step in building natural human-machine interactions

Christophe Tauziet
Dec 15, 2016 · 6 min read

Earlier this year, I joined the Social VR team at Facebook. The goal of our team is to explore how virtual reality can help create new rich and meaningful ways of connecting with the people you care about.

In October, Mark Zuckerberg gave a live demo of some of our recent explorations during the Oculus Connect keynote.

VR is still very much a fragmented and diverse medium with dozens of different headsets and input controllers. Our team decided to focus our work on the recently released Oculus Touch controllers. We recognized early on that bringing full-hand capabilities into VR through these controllers creates unique opportunities for meaningful social interactions.

In the next few years, VR is likely to grow beyond what we know today, but there are still valuable lessons for designing for what we already know. Following are a few lessons learned as we created natural and comfortable interactions for hands in VR. At the rate of the discoveries we’re making each week, some of this might be irrelevant or silly in just a few years—but we hope it can help everyone continue to make progress.

Creating natural interactions

I sometimes describe designing for VR as being dropped into the ocean and not knowing how to swim. You don’t really know where to go and don’t always know the way to move forward. The only things that keep you from drowning are re-using some of the problem solving tricks we use in product design to solve complex problems, looking for any opportunity to grow every new skill useful for this medium and helping each other out. Coming from a traditional product design background — where most interactions have many established and proven patterns — the most effective way I’ve found so far to make progress and learn in VR is through trial and error.

When it comes to designing for Oculus Touch, we realized early on that anyone new to VR had to make two radical shifts in their thinking. The first was the immersive aspect: looking around, having a feeling of presence and distinguishing the virtual space from the physical one. The other was how this new type of input controller suddenly gives you virtual hands and lets you interact with your surroundings.

When designing for control, it’s important to look at the most natural and intuitive ways of doing a specific action. Doing so allows you to create experiences that are approachable and delightful: whether that’s reaching out, picking up objects, interacting with a 2D panel, something in the distance, and so on.

Through countless demos and usability studies, we discovered that relying on controller buttons had a few drawbacks. People who weren’t used to video games had trouble finding the buttons without being able to see the controller. And using invisible buttons could potentially break the experience by reminding you your virtual hands weren’t real. This problem was even more obvious in a social context where people were trying to perform actions while having a conversation with someone else in VR.

Mark Zuckerberg using Oculus Touch in our Social VR demo.

We decided to always make the default interaction something that can be done with the virtual hand itself, and doesn’t require the use of any of the front-facing buttons of the controllers. Picking up an object and holding it is done by pressing and holding the grip. Interacting with a 2D user interface is done by simply touching it. Selecting something in the distance is done through pointing at it with your index finger. Scrolling through a list is done by either grabbing the list with your hand and moving it up and down or by grabbing and moving its scrollbar.

We’ve also learned how crucial visual, audio and haptic feedback can be in creating natural interactions. When approaching an object with your hand, giving feedback such as the object enlarging or brightening, or seeing your hand adapt by opening or otherwise adjusting its pose helps understanding that object can be interacted with. When touching the object, getting haptic and audio feedback adds to the immersion and creates a connection between the virtual and physical hands. Think of the sound your hand makes when you clap, or the subtle sensation of picking up and setting down your phone.

Designing comfortable experiences

While using your hands to interact with a virtual world is incredibly exciting and immersive, it presents some physical challenges for designers to consider. Here are some of the things I’ve learned:

  • The higher the person has to raise their arm to perform an interaction, the faster that interaction should be in order to avoid fatigue.

More Resources

Hopefully these insights help you when considering designing for hands in VR. We’re still in the early days, and as we make progress in this journey our design process also evolves and improves. Today, we’re excited to release a set of Virtual Hands we’ve created that can help designers mock up hand interactions for VR in traditional design tools. And if you’re curious about some other aspects of building a Social VR experience, checkout Mike Booth’s talk at Oculus Connect.

Virtual Hands for mockups

Check out more stories and resources from the Facebook VR design team at facebook.design/vr.

Facebook Design

Designing for the global diversity of human needs.

Thanks to Gabriel Valdivia, Jasmine Friedl, Arthur Bodolec, Tanner Christensen, and Jonathon Colman

Christophe Tauziet

Written by

Leading design for Facebook Spaces - Social VR at Facebook. Previously Parse & Apple.

Facebook Design

Designing for the global diversity of human needs.

More From Medium

More from Facebook Design

More from Facebook Design

Designing Against Misinformation

More from Facebook Design

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade