A fundamental guide to user testing in Virtual Reality
Handy facts to keep in mind when testing on any VR platform
Despite VR being widely available these days, there aren’t many guides or articles on how to set up user testing for it. Since 2017, we have been experimenting a lot with VR technology at Ostmodern, so decided it would be helpful to share some of our best advice for testing in terms of: recruiting participants, setting up the testing room, and ways of collecting insights for usability in VR.
Three points to remember about recruiting participants
- Although VR equipment is well distributed, it is still quite expensive and not yet universally adopted. Because of this, those who use it today may not be entirely representative of a future, potentially more mainstream, audience. For testing, it is important to recruit both current users and participants who are considering adopting the technology, to ensure a sample size that is more representative of the present and future market.
- In practical terms, be mindful of how best to accommodate participants who wear glasses: the headsets are generally too uncomfortable for them and these are unable to fit wide frames. Warning people about having to wear contact lenses early on in the process has saved us from sending disappointed participants away.
- Medical and psychological conditions must be taken seriously into consideration. Here is a quotation from the Oculus Rift user manual that we draw on and modify in our introductory protocols and consent forms with participants. It may seem strong but it’s in fact meant to warn participants about the risks:
“Content viewed using the product can be intense, immersive, and appear very life-like and may cause your brain and body to react accordingly. Certain types of content could trigger increased heart rate, spikes in blood pressure, panic attacks, anxiety, PTSD, fainting, and other adverse effects.”
Set up the room and devices as you would for a person who is unsighted
- While using a VR headset, users are not be able to see or easily navigate their physical surroundings. Avoid making unexpected contact with them and clear the test areas of objects on which people can trip, stumble, bash themselves, or hurt themselves in any way.
- If testing with a stationary VR set (which does not require full body movement), treat it as if it were a normal desktop testing exercise and have the participants sit down for the session. There is still a risk that the participant will move unexpectedly so keep the area clear of unnecessary objects and sharp corners.
- There is an ongoing debate about allowing swivel chairs in testing sessions as they make it easy to explore 360º environments. We suggest avoid using them: these chairs may disorient users, which in turn can be dangerous. Unless they are representative of the space in which the participant will use the product, don’t risk it.
Consider the placement of cameras and other equipment
- As a general rule, we always record a mirror image of the participants’ perspective, the controllers’ inputs and a face/body camera angle.
- Placing the camera is challenging; we don’t want to miss what happens in blind spots but there isn’t always the budget for 6 different camera angles. Our best compromise so far has been to set up two cameras on opposite sides of the participant. For smaller budgets, one camera can still do the trick if placed on the ceiling directly above the testing area.
- In case you are testing on mobile/non-stationary VR, keep the testing area narrow so that participants don’t disappear from the camera’s view.
- Offer wipes at the end of each session, as well as changing the headset padding and drying controllers for the next participant — there’s a high chance users will sweat.
- If testing on smartphone-driven VR, be prepared with spare smartphones and chargers. Testing a 360º video experience on Samsung VR drains about 10% off the battery in five minutes.
- Be conscious of power sockets and cables: one of our testing participants managed to turn off the power supply of a VIVE base station.
How to collect insights in an immersive environment
When testing with participants who are new to VR, it’s important to remember that the experience of diving for the first time into virtual reality can be so impressive that they are often too astounded to give objective feedback.
To overcome this issue, allow more session time with them, show them an easy experience first and then confront them with the actual prototype that needs testing. We usually like to use Google Earth as it’s a familiar piece of functionality that gets participants quickly used to various notions of space.
There are two main methods we use for finding out information from participants:
- The first is to ask participants to perform one task that should take no longer than five to ten minutes to accomplish, and pose them questions afterwards. This practice doesn’t interrupt their workflow and allows us to test smaller tweaks made in a product.
- The second approach consists of questioning participants through the VR headphones. Once again, this doesn’t interrupt their workflow but it also allows more vivid feedback. We use this method when we explore bigger concepts and ideas.
During long testing sessions, however, participants often forget their physiological and psychological limitations and this may require more precautions on our side when asking questions. Here at Ostmodern, we have decided to track not only their body language and words but also their heart rates.
From being chased by cold-blooded zombie killers to having a phobia of whales, we have seen a number of variations in heart rate in unexpected situations. A heart rate won’t tell you the full story, but combined with other observations, it can point out when a participant needs a break, or when you should prompt them about how they’re finding the experience.
On high intensity products, it is important to be aware of the effects that an experience can have on different people.
In a gaming test, one participant moved more frenetically than any other. While an average heart rate would increase by 30bpm during major actions, his increased by 50bpm. In this case, we refrained from asking him questions during the test and favoured post-task prompts.
On lower intensity products (e.g. Netflix VR, theBlu), participants tend to be more relaxed and it is easier to prompt them when their heart rate goes up without being too disruptive of the experience. This allows for an easy way to uncover frustrations and pain points that users encounter.
Testing as a signpost to the future of VR
Some of these methods will evolve and adapt to new VR hardware and software in years to come. The best guidance we can give is to think about your testing as rigorously as possible: as on any other platform, always consider your product’s purpose and what specifically you need to learn. Doing so makes it easier to decide on recruitment, set-up, and the testing methods to use.
In the meantime, keep in mind that VR is maturing as a tool with a set of associated practices and behaviours, as websites and smartphones have in recent years.
This is is why testing in VR is such an exciting research area at the moment: while users are still adapting to it, there is much space for innovation. Testing is the best way to map which inputs and functionalities so far provide a memorable and intuitive experience to the wider market.
Thank you for reading!
If you like this article, give us 50 claps. If you really like it, let us know why. Leave us a comment with any question or opinion you might have on the subject.
You can reach us on Twitter @Ostmodern to continue this and other discussions.
Follow our publication for more stories on tech, design, video and other interesting topics.
