Designing for Hands in VR
The next step in building natural human-machine interactions
Earlier this year, I joined the Social VR team at Facebook. The goal of our team is to explore how virtual reality can help create new rich and meaningful ways of connecting with the people you care about.
In October, Mark Zuckerberg gave a live demo of some of our recent explorations during the Oculus Connect keynote.
VR is still very much a fragmented and diverse medium with dozens of different headsets and input controllers. Our team decided to focus our work on the recently released Oculus Touch controllers. We recognized early on that bringing full-hand capabilities into VR through these controllers creates unique opportunities for meaningful social interactions.
In the next few years, VR is likely to grow beyond what we know today, but there are still valuable lessons for designing for what we already know. Following are a few lessons learned as we created natural and comfortable interactions for hands in VR. At the rate of the discoveries we’re making each week, some of this might be irrelevant or silly in just a few years—but we hope it can help everyone continue to make progress.
Creating natural interactions
I sometimes describe designing for VR as being dropped into the ocean and not knowing how to swim. You don’t really know where to go and don’t always know the way to move forward. The only things that keep you from drowning are re-using some of the problem solving tricks we use in product design to solve complex problems, looking for any opportunity to grow every new skill useful for this medium and helping each other out. Coming from a traditional product design background — where most interactions have many established and proven patterns — the most effective way I’ve found so far to make progress and learn in VR is through trial and error.
When it comes to designing for Oculus Touch, we realized early on that anyone new to VR had to make two radical shifts in their thinking. The first was the immersive aspect: looking around, having a feeling of presence and distinguishing the virtual space from the physical one. The other was how this new type of input controller suddenly gives you virtual hands and lets you interact with your surroundings.
When designing for control, it’s important to look at the most natural and intuitive ways of doing a specific action. Doing so allows you to create experiences that are approachable and delightful: whether that’s reaching out, picking up objects, interacting with a 2D panel, something in the distance, and so on.
Through countless demos and usability studies, we discovered that relying on controller buttons had a few drawbacks. People who weren’t used to video games had trouble finding the buttons without being able to see the controller. And using invisible buttons could potentially break the experience by reminding you your virtual hands weren’t real. This problem was even more obvious in a social context where people were trying to perform actions while having a conversation with someone else in VR.
We decided to always make the default interaction something that can be done with the virtual hand itself, and doesn’t require the use of any of the front-facing buttons of the controllers. Picking up an object and holding it is done by pressing and holding the grip. Interacting with a 2D user interface is done by simply touching it. Selecting something in the distance is done through pointing at it with your index finger. Scrolling through a list is done by either grabbing the list with your hand and moving it up and down or by grabbing and moving its scrollbar.
We’ve also learned how crucial visual, audio and haptic feedback can be in creating natural interactions. When approaching an object with your hand, giving feedback such as the object enlarging or brightening, or seeing your hand adapt by opening or otherwise adjusting its pose helps understanding that object can be interacted with. When touching the object, getting haptic and audio feedback adds to the immersion and creates a connection between the virtual and physical hands. Think of the sound your hand makes when you clap, or the subtle sensation of picking up and setting down your phone.
Designing comfortable experiences
While using your hands to interact with a virtual world is incredibly exciting and immersive, it presents some physical challenges for designers to consider. Here are some of the things I’ve learned:
- The higher the person has to raise their arm to perform an interaction, the faster that interaction should be in order to avoid fatigue.
- Raising your arm high enough that your elbow is no longer in contact with your body can be tiring.
- Interacting with a moving interface (for example, a UI attached to the secondary hand, like a watch) can be challenging and should be primarily used for quick actions.
- Some people have trouble visually reading depth, and may need some time and practice in order to learn how far they have to reach out to manipulate objects in VR.
- When sitting, reaching out towards objects that are too close to your virtual body can be challenging.
- Hands that visually look realistic can be creepy. They give people a feeling of being inside someone else’s body. They can also break the immersion when those same hands go through what appears to be a physical object, like a table.
- A familiar object often communicates how it should be picked up, held and used. For example, people expect to use a gun-shaped prop to aim at things.
- A physical object may need to have different physics when being held. If you’re holding a pen and go through the table, you probably don’t want the pen to hit the table and bounce, but rather go through the table with your hand.
- People lose track of their real world physical surroundings when in VR. Hand gestures can result in them hitting a wall or throwing the controller away.
- Hand gestures are inherently social. When your friend is interacting with virtual objects in VR, having a visual indication of what they’re interacting with removes the potential awkwardness of seeing their hands move in mid-air with no understanding of what they’re performing.
More Resources
Hopefully these insights help you when considering designing for hands in VR. We’re still in the early days, and as we make progress in this journey our design process also evolves and improves. Today, we’re excited to release a set of Virtual Hands we’ve created that can help designers mock up hand interactions for VR in traditional design tools. And if you’re curious about some other aspects of building a Social VR experience, checkout Mike Booth’s talk at Oculus Connect.
Check out more stories and resources from the Facebook VR design team at facebook.design/vr.
