VR Usability. Part 1
Why and how to test VR apps
Article by Olga Ivanova, contact via olga@vroxygen.com
VR, AR (XR) User Testing and Usability Studies. Contact Us:
VR Oxygen Home
Testing XR (VR, AR) Apps | Sharing the Findings of VR Design | Learn more!
vroxygen.com
When building a VR app everyone wants to get a lot of downloads and a 5-star rating. How to make sure that Your VR experience is comfortable, delightful and the one that people want to have, and really love it? Below is the plan to conquer the unknown and have a win-win situation for both, your XR app and players of this app.
- Answering the main questions of VR usability testing: why test, when, what to test, with whom and where to find those who desperately want to play your VR app.
- Usability testing methods, including user testing and compliance review, which are very beneficial.
- Some of the key findings. How to create hypotheses based on what you learned. Some tips for creating a comfortable VR experience.
Why Test?
The first question is “Why” is it important to test. First of all, we want to test in order:
– To evaluate a VR app, to see what doesn’t work as intended and get some insights into how to improve it
– To help find possible bugs
– To avoid cancelling and throwing away the whole idea and to prevent rebuilding, at least on a big scale, it helps to be more careful with the budget
– User Testing provides and opportunity to identify problems along the way, so there is enough time and resources to iterate
– You can also use user testing to generate new ideas. It can be a tactical user research and tests, for example, to understand what to build next.
In general, when testing there can be plenty to discover such as:
– Do ppl understand how to use it and what’s the learning curve for them
It shows how do different demographics use it and if patterns appear (what they do or try to do)
– Do people enjoy the app or not, would they recommend it or play again themselves? Was is comfortable? What was their favorite part?
– Testing is important to ensure that the design decisions are right and are based on business goals and people who will use this VR experience. It may also help to secure the longevity of vision and make sure the app idea works in the long run.
– And, also building for VR is not the same as building for desktop, so there can be unexpected moments.
– One of the other goals of User Testing is to uncover people’s Mental models –what people think about something, what they assume (they didn’t try, they don’t know abou VR or certain game or app, but they already assume and expect something — those are large parts of what may hold some people back from VR at the moment, for example.
And, finally, when you are aware of it all, you can build a better VR experience to make people happy, and you will get more downloads because you’ll know what the potential users want and what they like.
When to Test?
Next what is crucial to consider is When to start testing?
It should be done early on and the sooner — the better.
When a first scene draft is created, it’s a good time to start testing. There is no need to polish or create a detailed prototype in order to test it.
You can test drafts (not interactive paper prototypes) to preview scenes, paper VR prototypes (skyboxes), storyboards-to-VR, early blocking (objects’ locations), test some small parts to check the color pallette, scales, locations and approximate distances. It’s easy to test rough volumetric prototypes (greyboxing) to see interactions. Just throw some grey boxes (cubes, planes, spheres) into Unity and use them as objects. Like designers create wireframes for websites and apps, here we can use simple forms for any object.
Don’t forget to test regularly hroughout the whole development process, from sketched and paper VR prototypes to 3D volumetric early drafts and more detailed prototypes.
Test at all stages of VR product development. When a 1-st draft is created, it’s a good time to start testing. Preview it in VR every time you make a change. Make a build for Android or iPhone to test in Cardboard VR Viewer if it’s mobile, or to test in Oculus which is especially convenient.
Test small parts (colors, interactions, scales, proximity (distances), brightness, contrast) to make sure the experience is comfortable.
The importance to test the experience in VR is essential, because it always feels very different from desktop. Even very rough small prototypes can confirm the comfort zones, locations and scales.
What to Test?
Next, it’s important to define What to test, which part of a VR experience should be tested.
Let’s say you already defined why you want to test an app. For example, you want to find out why people quit after certain part of the experience or after some time. You want to increase playtime or to ensure people continue playing at least until certain part or level.
To make it happen, depending on the app itself, you may want to test what can cause game interruption, there can be a variety of reasons — from bugs in game, in the code, that simply make the game laggy or just crush the game in the middle of it where you think people quit, or make it too slow to load or proceed, but can be that it’s an uncomfortable VR enviroment that causes motion sickness, eye strain etc. It might be a contrast issue and that people just don’t notice the UI.
Or the reason can be that people simply don’t know how to proceed further in the game, they get confused and eventually quit, so people get stuck without any clue of where to go or how to go there.
You may want to test certain pieces of an app, for instance, an interaction, a scene if it’s confusing, colors if they are too bright, or any other aspects.
First, it’s important to know what you want to find out and how it will affect the VR experience after you get the answer to your questions. Will it help? Then you should test it.
Whom to Test With?
Now, as we know the answers to all the previous questions, let’s see Whom would it make sense to test with?
The easiest way to test at the very beginning of design and development is to test yourself — check in Cardboard, Oculus or whatever platform you work with. For sure, involve all your team into it.
It’s also important to test with people with domain expertise in the industry type that your product is in, for example, medicine. If you are working on some medical-related VR app, you would need to test it with people who have an expertise in this topic. You can also show your VR app to people who are experts specifically in the VR industry.
And, of course, test with people from your Target Audience (TA), otherwise it all might be useless. Test with those whom this app is intended for.
How many testers to select?
5 should be enough to test usability issues, some features, but 10 wouldn’t hurt either.
To study demographics you would need more people (example — people aged 15–17 y.o, or ppl who play certain type of games).
When testing, don’t forget to compare New users (people who have never used VR before, or may be just tried once, but has no knowledge about it) and experienced VR users (those who use VR regularly and most likely own a VR device, or even develop for VR). Of course, it all depends on your TA, but usually the most wanted testers are New Users.
You can also consider experienced VR users but who are new specifically to your app. To recruit someone from VR audience seems a bit easier. Testing with new VR users is recommended even though it’s challenging, but get creative, offer something, if not money — may be something relative to your VR app, early access, or jut see what people from your particular TA could be interested in and try.
There are definitely, some pros and cons with new and experienced VR users. Experienced users can test your app remotely, because they have all the hardware. But they can be already used to various locomotion techniques, so it’s hard to test motion sickness, for example. They may also know how to use some features because they have a previous VR experience.
Another thing is, that in the case of testing with new VR users you would need to carry your target device or invite people to your office, or cafe or co-working place in order to conduct testing. On the other hand, they will have a fresh view on your app and most likely their opinion will be unbiased.
Where to Conduct Testing?
The next challenge is where to find all these people who desire to playtest your game?
Well, it might be not so easy, but you can try recruiting people on Reddit or some VR related groups and events. You can also attend non VR related expos where it will be suitable to showcase and demo your VR app.
It’s also important to consider the context and User Journey — why and where would people want to use your VR app? Is it intended for study or for leisure and entertainment? Where will people want to use it? Probably, not in the pool, but you get my point that it can be very different.
User Journey represents the full end-to-end experience including the emotional part, motivations, actions, interactions, touchpoints, emotions and feelings. For example — where and why a person would want to use your VR app? What would trigger this wish?
Another detail I want to mention is that the way of finding users may be affected by the type of VR build itself. If it’s webVR, it’s easier to share and to preview, it’s clear though that the testers will be experienced VR users. Since there is no need to download, you would probably get more responses becasue it’s very accessible, you can just use your browser. So get creative and enjoy the process!
These are the main questions to answer before actually start testing.
Below is what will be covered in the Part 2:
Or watch a video (full 3 parts combined):
Read Part 2 here
Read Part 3 here
Please, contact via olga@vroxygen.com
VR, AR (XR) User Testing and Usability Studies. Contact Us:
VR Oxygen Home
Testing XR (VR, AR) Apps | Sharing the Findings of VR Design | Learn more!
vroxygen.com
Read more of my articles:
Check my Case Studies:
Visit vroxygen.com
Watch the Usability Study video:
Or Test your VR app Yourself!
