How to design a realistic usability test
Real life isn’t just one place. It’s also hardly ever like a usability testing lab.
There are a variety of different usability testing approaches — from high-tech usability labs to remote testing or quick coffee shop sessions.
Although the choice of methodology and setup for usability testing depend on the research objectives, the users and the product itself, there is no doubt that it’s important to create a scenario that is as close to real as possible. Here’s why.
Why you should be realistic.
Designing for mobile implies that we are designing for multiple different environments and contexts. We hardly ever use our phones, tablets or smartwatches in quiet, neutral spaces like conference rooms or usability labs. Yet, for some reason we often test our prototypes exactly there.
Consider the usability testing lab benchmark: There’s a dedicated space with several computers set up, running monitoring software to record the screen. There will be a camera to record the user, while a team of testers and observers are watching through a pane of one way glass or via video stream from a separate room. The more sophisticated setups also include eye tracking and emotion recognition technology, as well as tools that measure a user’s brain activity.
While all of this can get us interesting (and controlled) insights into a users’ attention, emotional state and level of cognitive engagement, a high tech science-like setup may cause users to be intimidated or uncomfortable, creating a behavioural bias. Also, in reality, modern humans spend most of their time in busy, messy environments, battling with imperfect devices or connections and interacting with products while simultaneously attending to other things.
Considering divided attention, device setup and environmental factors, and including these as parameters in your study can identify issues that formal testing wouldn’t.
Here are 5 pieces of advice on how to make your testing environments more real.
1. Devices, operating systems and set-up
When inviting users to your testing space, make sure you test with the devices, operating systems and accessories they are used to. It may look great to have a brand new iMac in your lab, but if it’s not what your users are using in real life, you may end up measuring factors related to unfamiliar technology, confounding the real product insights you are actually after. One way of dealing with this, is by asking your participants upfront which devices (tablet, mobile, desktop), accessories (mouse, keyboard etc.), operating systems and browsers they are most likely to use the product on, so you can provide them with the respective setup. Alternatively, participants can bring their own devices, which enables you and your team to see how their desktops or phones are set up, what other apps or browser extensions they use or how they interact with it (e.g how they use tabs). Generally, insights into how users work with their own machines are very valuable for discovering user pain points and to generate ideas on how to fix them.
2. Different context, different intent, different device
Most people are used to using multiple different digital devices, depending on the situation or environment that they are in. In addition to different devices though, different environments also prompt different tasks. Consider a user of a company intranet: She might be accessing the system on her phone during her morning commute to the office, checking on the status of her projects, viewing notifications and catching up on messages. The factors influencing these tasks are slow, intermittent internet connections while on the train and interrupted use of the device/app when changing trains. When arriving at the office, she accesses the Intranet again, this time from her desktop computer, downloading the most recent analytics report, or creating and uploading content for a new project. The environment is quiet, enabling focused work with a fast wifi connection on a desktop computer. The point is, that the use of some product features is device-specific as well as context-specific, i.e. users may only engage in certain tasks within a given environment, using a specific device. It’s worth understanding and keeping these in mind when designing your usability study so that you can focus on the most likely scenarios.
3. One user vs. multiple users
While most moderated usability testing sessions are conducted with 1 user at a time, some products are inherently multi-user focused, e.g. multiplayer games or social networking apps. Testing them in isolation with 1 person, may only give you one side of the story and thereby miss the intentionality and interaction aspects that play a role in real life. Having said that, not only multi-user products may require you to consider more than one user. Although some tasks may be completed by a single user, a product experience may be heavily influenced by multiple secondary users. An example of this is booking travel. It usually involves a primary user booking the trip (i.e. completing the purchase), after searching and discussing different travel options with one or more secondary users. Think of groups of friends, couples or families which will all want to take part in the decision making process. Involving them in a testing session and observing the interactions (e.g. sharing, creating lists, receiving notifications), can uncover user goals and behaviours that you wouldn’t know about otherwise.
4. Simulate environmental factors
Real life happens in a variety of places; a kitchen, on the couch with the TV on or in a car. If you can’t meet the users in the environment the product is typically used in, but you have a usability lab or a dedicated space for testing, one option is to transform it into a simulated version of that environment. Although it may be obvious that the environment isn’t real, it gets people in the mindset of the product’s context of use, shifting their thinking more towards the way they would be thinking and behaving in that actual environment. Moving away from an unrelated academic lab space can improve the quality of the user feedback by yielding more unique insights, drive better conversations and prime the user to think about potential issues or complications with the product. Apart from the physical space, the user’s mental state, e.g. extreme fatigue, the influence of alcohol or stress are important factors that contribute significantly to an experience. With these factors being more difficult to replicate in a lab (aside from ethical considerations) they should at least be known and considered when designing and analyzing a testing session.
5. Simulate the environment with VR
Another way of simulating a real life context without having to re-decorate your usability lab is by leveraging VR technologies to build virtual prototypes of an environment. While the application of VR to test store layouts in retail has recently become quite popular, translating this idea to usability testing hasn’t received much attention yet. It’s an exciting one however, holding the potential to bring immersive experiences with multi-sensory and interactive features into any regular space. Especially mixed reality, where virtual objects are integrated and mapped into a physical view, could be very useful when testing extreme conditions such as mountaineering or rescue apps, airplane dashboards as well as medical applications that are used during surgery. While this is still new and emerging technology (and it doesn’t come cheap), in the future this holds great promise to be applied to a variety of different scenarios, including usability testing.
TL;DR?
Here is what’s important in short.
Test the experience in an environment and context as close to real as possible. If it’s likely to take place at a desk, test it at a desk. If it’s likely to take place on the top of a mountain, test it there. This way, you can take physical or mental factors associated with a space into account when designing and testing the experience. Don’t test in a controlled UX testing lab and expect to learn about an experience in a totally different space. If you don’t have access to that space, simulate it as best as you can. ✌️
Looking for a UX design agency to partner with?
We’re on a mission to empower organisations with strong, informed design. We also do moderated usability testing (with realistic set-ups)! Get in touch.
Be sure to follow our journey and connect with us on:
Website | Instagram | Dribbble | Twitter
