Testing the User Experience of Virtual Reality
Notes from Guerrilla Usability Testing for Oculus Rift and HTC Vive
Last week, I came across a demo of the last Oculus Rift VR controllers at a mall in San Francisco. There were five people ahead of me in line, and the researcher in me couldn’t help but observe and analyze their behavioral patterns.
As a user, I’ve been fascinated by virtual reality’s user experience since the first time I tried it. Even though that early model had rough 3D graphics and stories that didn’t make much sense, VR changed me completely. The experiences were always so engaging that I felt as if I’d really been in the virtual experiences I lived.
My curiosity about this effect is the reason why I go to every VR demo that I can find. The last time I attended one, I started identifying some patterns in the way users learned to use the controllers while I was waiting in line.
One of those patterns had to do with the user’s previous gaming experience: experienced gamers would learn faster how to play.
I first noticed it because two of the users in line were brothers, and they were joking around about their abilities. The first person that played didn’t have gaming experience, while the other considered himself a heavy XBOX player. I further verified my idea with the next users in line: one 33-year-old male said he plays a lot of video games in PS4 and has had previous VR experience. The last two users had experienced video games but weren’t serious players.
Let me show you this and other patterns I identified by observing five users try the new controllers and this VR experience.
Oculus Rift Pain Points
- People who did not have gaming experience prior to trying Oculus had trouble using the controllers and getting used to the new visual experience. They expended more time learning and less time actually playing.
- For three of the five users, once they were within the VR experience, it was very difficult to identify which buttons to press according to what the screen was suggesting.
- The tasks that implied pointing with a finger to choose a category were the most confusing. The majority (4/5) of the users struggled to identify which button to press first, and where to point next.
- Even though almost all of the games had an onboarding tutorial, every user still needed outside instructions from a facilitator before starting to play.
Exploring new VR experiences
The following week, I met Ilya Druzhnikov from Exit Reality: a beautifully designed truck that offers HTC Vive VR experiences for 10 minutes per person. They offer a unique environment inside the truck, providing a safe space for users to experience virtual reality with human guidance. Users get to test the newest games using the latest VR technology.
Two weeks ago, Exit Reality invited me to join them at ‘Off the Grid’, at the Computer Science History Museum. As a researcher, it was a priceless opportunity to observe people of all different ages experience VR through HTC Vive for the first time.
During that night, I’ve watched 15 people experience VR for the first time. After a Star Wars event at the Computer Science History Museum, families dressed in Jedi costumes would stop by the truck to experience space attacks, archery training, mountain hikes or sea life adventures. All of them enjoyed it and were totally moved by the experiences.
I also found some patterns in the way people learned to use this device and its controllers:
HTC Vive Pain Points
- Overall, people learned how to use the HTC Vive controllers more quickly than with Oculus Rift. They have only two buttons — one in the back that’s usually a trigger, and one for the thumb for scrolling. In certain VR games, the left and right hands do different things. For some users, recognizing which one does what was difficult.
- Even though the HTC Vive learning curve was less steep than that of Oculus Rift, users still needed outside instruction to learn how to use the controllers.
- Selecting objects was also the most complicated task for first-time users.
- In HTC Vive, when you have to press a button in VR, you have to do it with your hands and body. For example, when turning on a light switch, you have to do the gesture with your arm pointing the switch. Zero users expected to have to press something with their arms, they thought that doing it with the controller could complete the action.
- All users needed outside guidance to learn how to select certain things in the VR experience; although once they understood it, it was very easily reused.
- I found the same pattern with gamers and non-gamers. Users with experience in gaming needed fewer instructions — just a short tutorial and they were in.
- Another issue some participants pointed out was regarding the sound: when users weren’t able to listen properly to something interacting with them (for example, in the game Space Pirate, the enemy ships shooting behind them), they couldn’t notice the interaction unless they were looking directly at it. In such interactive games it’s important for the creators to understand which senses are activated during each interaction and decide when hierarchies are necessary. Users needed more than simply the visual reference to understand their environment.
At the same event, I also talked with Yoni Koenig, Co-Founder and CEO of Exit Reality. He explained to me that it’s better to show passive experiences like The Blu or Invasion to first-time VR users. After they’ve become familiarized with the environment, it is easier to teach them how to use the controllers in more interactive experiences.
I was able to verify this observation several times: Users that experienced a passive experience first, would take less time to understand how to play an interactive game.
Lessons Learned
Many tech professionals are trying to break into the VR space. As we do, we need to build design best practices and advice specific to this type of user experience, e.g. designing to minimize motion sickness. Some VR experiences are more intuitive, while others are messier. The difference lies in how each experience is designed.
Next Steps
Based on my observations, research and experience using VR, I’ve laid out the following suggestions for optimizing the user’s experiences for VR:
- Minimize controller buttons: Design as few buttons as possible, and make sure each button is unique so the user does not get confused.
- Unify actions done with a controller: Make the thumb’s action to scroll and the index finger action to press. This way it’s much easier to differentiate which action the interface is asking the user to do at all times.
- More storytelling: No matter how cool the controller is, VR experiences need more storytelling and context for users to be onboarded successfully.
- The experiences are visually immersive, but if the sound doesn’t support it, it becomes boring and messy. Sound needs to be high quality to provoque a full immersion.
- Practice exercises in onboarding: Include exercises in the tutorials to help users learn how the interactions work. These exercises should be implemented with visual and audio instructions. Oculus Rift had a good onboarding process but lacked feedback about how well you’re doing. Also, regarding the complexity of the controller, it wouldn’t harm doing a longer tutorial.
Although both systems present usability issues and VR still has a long road ahead to improve, every single user that tested the VR games was fascinated. One middle-aged woman came back twice and left saying, “Can’t wait to get back to the game!”
While there is a lot to improve and learn about it, I have no doubt that virtual reality is about to change everything. I’m glad that there are people like Yoni, Carla, Jorge, Ilya and Khaleb that are working to bring VR to the people and let everybody experience what’s about to change our everyday lives.
From my side, I’ll continue improving my observations and measures in the performance of different VR experiences. My goal is to create a framework for usability testing in the VR space. Follow me to receive updates on my next review of Google Daydream View.
I’m a independent User Experience Researcher, I am not affiliated with Oculus or HTC Vive, nor Exit Reality. Special thanks go to my editors ❤
If you’re designing for VR and would like to conduct user testing, feel free to drop me a line at bceballos@tradecrafted.com or @benjibel.
