VR Usability. Part 2
Why and how to test VR apps
Article by Olga Ivanova, contact via olga@vroxygen.com
VR, AR (XR) User Testing and Usability Studies. Contact Us:
VR Oxygen Home
Testing XR (VR, AR) Apps | Sharing the Findings of VR Design | Learn more!
vroxygen.com
Let’s finally get to the testing! Let’s check several methods that could be useful to practice.
There are a lot of usability methods and techniques, but from the testing experience, the following 4 are the most useful and 3 of them are widely applicable.
They are:
1. Heuristic Evaluation
2. Guerrilla User Testing
3. In-person and Remote VR Testing
4. VR Analytics, Heatmaps
When selecting a test type, it is crucial to select wisely based on:
– what is being tested, platform etc (Cardboard, Oculus etc.), what type of experience it is, what TA it has
– At what stage the experience is at the moment of the test,
– Number of users. For example, for VR Analytics it’s essential because it won’t make any sense otherwise if an app has too few users yet.
When it’s all clear on those details, it’s easier to proceed.
1.VR Heuristic Evaluation
Let’s start with VR Heuristic Evaluation. It’s a VR Experience evaluation method based on experience’s compliance with certain VR principles (the “heuristics”). The heuristics should be selected carefully based on the app specifics. They should be applicable and appropriate for your VR experience for the VR experience.
It’s a great tool to use at a stage before testing with real users, before actually talking to people from your TA.
Evaluators should not be your end users, not yet. You can do it yourself, together with your team and preferably with these who work in the field your product is in (e.g., medicine, education), experts that were mentioned earlier.
The list of Heuristics can be found online and is called a VR Heuristic Evaluation Tool.
These heuristics were made based on the reasearch and observations, user testing, and on some additional valuable resources.
It’s a check list that helps to make sure that the major VR principles are followed. It’s not a set of concrete rules, but a flexible guide which everyone can apply depending on the experience being tested. So it’s more of a thought provoking material to help everyone check if everything was considered.
It’s a quick and inexpensive feedback. And you can obtain it early in the design process.
Below is the link to an online VR Heuristic Evaluation Tool:
2. Guerrilla Testing
Guerrilla user testing is usually a relatively simple and low-cost method that involves, let’s call it, a facilitator, approaching people and asking to participate in a test, showing them an app and asking questions. It’s usually very casual, and participants can be found practically anywhere from cafe to an expo — wherever you can locate your TA. And you can also select a location based on the User Journey.
It’s really easy to conduct Guerrilla Testing using a Cardboard since it’s portable and lightweight.
Definitely, for a VR application to be tested with real users, it should be a 3D VR prototype with some more or less defined environments and objects. It needs to have the main interactions, sound, some visual design but doesn’t have to be polished yet.
Attending various VR and non-VR expos and events in order to recruit the testers and specifically the first-time VR users can be helpful. It allows to demo and interview them in a casual setting, and definitely to observe them, observations play really big role in VR testing.
Guerrilla testing is perfectly suited :
– To conduct frequent tests throughout the design and development cycle to identify the usability issues
– To refine the experience
– To manage tests when access to the end-users is limited
3. In-person or Remote Study
Probably, the most productive and the most challenging user testing method.
How to conduct it:
First, you need to bring the testers into your office or wherever your VR setup is, physically, which adds some serious time and may be even money constraints. For in-person testing you would need a full setting — an office or a room with a full set up that includes:
– An HMD you are testing with (let’s say, we use Oculus)
– A camera for external recording or you can use your phone — to film the participant, their movements and reactions + sound recording.
– Since in VR we can not directly study the participants view, we should record a gameplay, you can use Oculus Mirror to mirror the VR experience from headset onto desktop, or combine it with OBS (or software by Nvidia) to record it, if you are testing with Oculus.
Below is a walkthrough example. The tester is thinking aloud and describing what he sees, what he does and what he thinks he should do. He asks questions, tells if he doesn’t know what he should do etc.
Record both, the gameplay and all actions the user takes, how they move, what they say. It’s really helpful to see the actual players to find out where they struggle, what they like and don’t like, when they are tired etc — it can be very obvious even when people tell the opposite.
One of the biggest measures of success is that an experience is comfortable and easy to navigate, the player should be able to find out what to do inside of VR. Not be endured to solve the riddle and complete the task, but make it with comfort and ease. Testing learnability will be essential.
(You can also include behavioral and Attitudinal test — what ppl do vs. what they say. — say they are fine, but are red or pale etc..)
– You may also collect data captured by a wearable device (e.g.,Apple Watch or Samsung Gear Fit) to record heart rate, blood pressure information (if needed) — if creating a meditation app may be, so you can approximatly measure if the person is relaxed or stressed.
Before conducting the test don’t forget to gather and document the demographics and info about the VR set up of a tester such as HMD, Graphics card, processor etc. Of course, we are talking about an experienced user tester in this case.
For a remote testing, also gather info about the VR setup of the tester such as HMD, Graphics card, procesor etc.
With new users, another important point is to make sure the issue the user encountered in not caused by the lack of the knowledge about how to use controllers. Make sure you explain it in before you test. So you can see that the users stuck because of in-game setting and behaviors, but not because they don’t know how to push the button inside of VR or how to grab an object. Don’t test the controllers, test your app.
Facilitator
And now I’d like to tell a little about the testing process. When in the Lab or anywhere you conduct the tests:
– Put participants at ease
– Introduce yourself, offer an explanation of the product and any equipment, make sure they understand
– Always Treat Users as experts
– Be impartial, ensure people can be candid with you and are fine to have good and bad comments. You can mention to the participants that you had limited involvement in the design/ dev. if you feel it’s necessary, so that they don’t worry about hurting your feelings and don’t show you are upset if people don’t like something you designed.
– Look for verbal, facial cues, observe. Note everything.
Pay attention to health, emotions, behavior.
– Have a script (at least in your head) and be prepared!
Note:
Health and Eemotions: People enjoy expressing positive emotions and excitement over VR, but they are less likely to admit to feeling sick, very often I find out about it later on, even next days after the test was conducted — they tell me. Sometimes they think it should be as such. Often they don’t know what they want or tell what You (as they think) want to hear, or think they might look weak, afraid if they do something wrong etc.
But you can find some things out, for example that they may be feeling unwell by observing: for example, they start sweating or they become red or pale, or they start brearhing heavily. Watch out for signs of discomfort to make sure they are ok.
Environment/ Surroundings: Pay attention to the surroundings, set up everything having enough space from the walls and other objects, ensure there are no obstacles or hazards near the participant.
Hygene: Prepapre wet napkins and you can use Surgical Dressigns (patches you can get in Walgreens or anywhere, they are cheap) as a protection, it sticks to Oculus’ foam.
Devices: Cool down Oculus and mobile phones (especially) between sessions, charge the phones.
Materials and Documents to Prepare:
Other consideration when testing is to prepare the documents needed for tests such as:
Tasks
People will forget the task as soon as they put a headset on. So spend some time writing the tasks to make sure they are not biased and the words are chosen carefully (avoid using the same terminology that is used on the interface, for example)…etc.
Think it over to create a set of the tasks which can be remembered and done within the headset. Tasks should be clear and short, easy to remember.
Important details to pay attention to:
Avoid clues and steps’ desriptions
Order of the tasks should not create clues either
Pay attention to the dependencies between the tasks
Something to stress on — don’t forget to ask the participants to sign the consent form, and may be other documents relative health if needed, or NDA etc.
More Documents to consider:
NDA
Consent Form samples can be found on the VR Oxygen website
You can use an app for your phone ot tablet, there are plenty, so people can sign electronically and have an e-mail sent to them.
Ethical Guidelines (review if needed) — for example, when testing with children or people with different abilities, check some psycholigical specifics to make sure you comply with all the possible situations.
Questionnaire in the end
And you are all set!
4. VR Analytics, Heatmaps
Another method you can use is Analytics.
Among the tools to measure performance and player engagement are Unity and YouTube, and they both have VR Heatmaps. But for YouTube the video must have 1000 views to enable it, and be 360. So it might be useful to publish some VR scenes on YouTube to receive more statistics. At the time of writing of this material, Unity required to have a Pro license. Check if it changed since then.
You can measure player engagement, where did people look at, how frequent they change position and use the interaction methods. Discover the most viewed areas of the experience and how long people are looking at a specific part of the video.
Read Part 1 here
Read Part 3 here
Below is what will be covered in the Part 3:
Please, contact via olga@vroxygen.com
VR, AR (XR) User Testing and Usability Studies. Contact Us:
VR Oxygen Home
Testing XR (VR, AR) Apps | Sharing the Findings of VR Design | Learn more!
vroxygen.com
Read more of my articles:
Check my Case Studies:
Visit vroxygen.com
Watch the Usability Study video:
TEST your VR app by Yourself!
