The value of user testing
Interviewed: Laura Lugo
Interview and Writing: Joshua Reddekopp
Editors: Stephanie Unruh | Brendan Swalm
Intro
Laura Lugo is one of the most contagiously positive and bright people I know. This might be part of why I was so excited when I learned about our shared passion for user testing. In this article, I’ve summarized a discussion Laura and I had after running a user testing campaign together for one of Vendasta’s Foundational Products, CalendarHero. We detail the what and why behind user testing, and cover what we learned together too.
I feel user testing is one of the most valuable contributions anyone can bring to their teams, we hope you’ll enjoy our overview of the practice below.
What is user testing?
User testing is a method designers utilize to test our ideas with actual users. The users aren’t being tested, they’re coming alongside us as designers to provide an unbiased analysis of a given solution.
Typically a user test is run in person by providing a participant with a prototype and a series of tasks to complete. The prototype can be high fidelity, meaning it closely models the visual and interaction design of live software, or it can be as simple as pieces of paper with drawings that the designer moves around to mimic software interactions. Often we’ll use a type of digital prototype that’s a blend of the two called a wireframe.
The key purpose and benefit of user testing is to evaluate the usability and value of a given solution relative to other options. I’m sure we’ll talk about its benefits more but what makes user testing so valuable is that compared to developing the actual solution, it’s a low-effort and low-risk way to increase the confidence of the work delivered to our development teams.
With user testing we can help mitigate risk and validate our software before a single line of code has to be written.
So, who should participate in a user test?
The best people to test with are real users or people who could be your users.
In a business-to-business scenario, like at Vendasta, it can be challenging to access users because our software is offered white-label (resold instead of sold directly to customers). One tactic we use to get around this is testing with new hires before they’ve been onboarded to our products. This provides a good proxy for the new user experience. How easy is it for new users to understand the product, or accomplish the key “job to be done”? These are questions new hires have helped us answer.
At the end of the day, any user testing is going to reveal insights into how humans interact with your design. A quick user test with a coworker in the hallway can really bring light to how clear or confusing a workflow is.
To summarize, the best folks to user test with are actual users, but there is a long list of other valid participants to help you illuminate the usability of your solution. Just make sure your participant isn’t too familiar with your work or invested in things going a certain way as this can bias your results.
Give user testing a try, we’re sure you’ll learn something!
If you had to explain why user testing is worth the investment, what would you say?
Running usability tests helps ensure you don’t lose money and time rebuilding things again. It’s much less costly to understand user needs early as a result of research than it is to understand them later as a result of feedback on live software. When you include usability tests as part of your software quality framework you solve interaction issues in advance and avoid unnecessary friction between your business and users.
The diagram below showcases this in the context of a larger project life cycle, it’s much cheaper and quicker to make changes during the prototyping phase. This is true for small changes, but the difference is exponential for bigger pivots. In a perfect world, we would test at every step of the way, but the gold standard will always be: test early and test often.
Design thinking has deep roots in the history of software development and its principles can be applied by more than just designers.
The first personal computer was created in 1976 by Steve Jobs and Steve Wozniak. Both were software engineers. In 1981 IBM launched its first computer on a larger scale. Again, only engineers had access to this computer. Only in the 1990s did the computer become more publicly available. Now even babies with access to computers — we are talking about billions of different users with a huge diversity in backgrounds, languages, perspectives, and needs. To help generalize industry lessons we have International Organization for Standardization, first published in 1991. Jakob Nielsen was also an engineer and is responsible for creating the 10 usability heuristics based on human-computer interaction (HCI). Designers understand human behaviour and bring that understanding into the software.
The original grandfathers of design were engineers. Design thinking is an understanding that the way humans interact with the world is often not aligned with cold machine logic and thus it must be considered and built for. In other words, even the original inventors of the computer knew usability was important as a central part of the design process.
Developers and designers strive for the same objectives — at the end of the day, we’re all responsible for the user experience.
Recently you ran a set of user tests on Calendar Hero, one of Vendasta’s foundational products. Can you tell me more about your process here?
Sure! I’ll start by noting that before I even ran my first usability test I carried out my own walkthrough and heuristic analysis of the product. This allowed me to generate some hypotheses on where users might be tripping up. Next, it was time to validate and this is where user testing came in.
One of the key metrics for this product is the number of meetings booked. This is a business need. Keeping this in mind, I can also understand where we are helping the user achieve this task and where we might be letting them down.
I want to learn what the real user workflow is. Users don’t always follow the intended path. Where are they getting lost? Stuck? Dropping off? These are all hugely valuable insights that contribute to our efforts of improving the user experience and ensuring business goals are met.
After reviewing my findings I discovered a large number of issues early in the proposed new user experience during the onboarding phase so we chose to focus there.
For a more detailed breakdown of the user testing process see the 101 resource by the NN group.
What did you learn from your first user test?
One of our most significant discoveries from user testing was that the product’s onboarding experience didn’t instill enough product trust in the user.
We learned that in order for users to feel comfortable utilizing an application to act as their scheduling assistant, they needed to first build a level of confidence and trust in the way this solution would display their schedule to their customers. It was very important to users that the solution allowed buffer times between meetings and didn’t double-book meetings or book outside of a defined window. While CalendarHero does allow for all of these affordances the existing level of onboarding was failing to get users to a point of clear education and confidence. It is often the case that a feature exists but isn’t surfaced to the user at the right time or way or is crowded out by other unrelated information.
At the end of our tests, we realized that the users did not complete their core “job to be done” of sharing a booking link because they weren’t sure of how it would work or appear to their end clients.
This is something that numbers don’t tell us. They’ll tell you what’s happening but rarely provide good insights into why.
How did you change the test after your first session? What does this say about the user testing process?
It’s important to test your test itself before any participants see it. I ran a pilot test with a friend to make sure all the links were working and the questions were clear, and to time how long the test took. We even cut out an entire section of our test and rescoped to ensure the test was focused.
Narrowing the test down was part of the iterative process. We needed to ensure the test was focused enough to fit the allotted time and not overwhelm the participants while also capturing as many insights as we could.
How do you get unbiased results from your user test?
It’s very important not only to set the user testing participants at ease but also to refrain from guiding them and biasing the test results. During the usability test, you need to be as invisible as you can. The idea is that you gather insights from the users testing the software, which makes it important to establish an environment where it’s clear that you are there to learn together. A critical aspect of this is using open-ended questions rather than leading ones. Below are some common test questions.
Why is it important to help participants be at ease and comfortable? Is there anything you do to encourage this?
Instead of framing it as a test, it’s an opportunity for the users to express their thoughts openly. Avoid technical terms like usability, prototype, test, experiment, interaction, software, quality, and so on. This is because, in the end, you are asking for help. You are asking someone, “Help me make this service or product better for someone like you.” “Can you help me understand if this product or service is good enough, and learn what you want to see here?” It’s very important to ensure the users know they are free to interact with the software however they like. They can click where they want and explore where they wish. This is all to help you capture the most reliable information that you can. I often make a point of reminding the participants that it isn’t a bad thing if they get stuck, it’s actually very helpful for our learning. The concept “We are testing the product not you.“ is key for your participants to understand.
Is there any advice you can offer to those who want to user test but are pressed for time?
I would say you have nothing to lose by giving it a try! Testing doesn’t have to be complicated. Testing a little, even in a simple way is better than not testing at all.
We also have the usability heuristics to help us improve the state of our software in addition to the study of human-computer interaction — make sure you start here, so that if you have time for testing you’re not capturing insights you could have easily caught yourself.
A “show instead of tell strategy” is often much more effective in drumming up support than cold stats. Do a user test, gather useful feedback, and let the results speak to their own value!
English is your second language; I can imagine this adds an extra challenge to running user tests with English-speaking participants. How did you face this challenge and what advice do you have to other folks in similar situations?
You must be like Batman — be prepared. Batman is my favourite hero because he doesn’t have any special powers. What makes Batman a superhero is that he is always prepared, he knows everything about everyone.
I also need to confess that driving usability tests is something that can trigger my anxiety and introduce stress because I know how important it is to be alert to the nuances: paying attention to not only the participant’s body language and what they’re doing, but also the words they use to communicate or hide an emotion. It’s really complex.
So when I say “be prepared,” I mean that if you know a lot about what you want to test and the outcomes that you need in the end, you will feel more confident and prepared to manage whatever situation arises.
Also, be kind and patient with yourself. I don’t necessarily do this — I am not a kind person to myself. This experience really threw that in my face. “Okay, maybe I should be nicer to myself.” Don’t make your own life miserable — we are humans and make mistakes. The great part about this is it gives us a chance to learn from the mistakes we make. Asking for help doesn’t make us weak or less than others, it actually makes us stronger.
To recap, be prepared, be kind to yourself, and ask for help when you need it.
Article Feedback
Thanks for reading, I hope you enjoyed it as much as I enjoyed putting it together
Please share your feedback here!
References
ISO 9241–11:2018. ISO. (2018, April 4). Retrieved November 4, 2022, from ISO 9241–11:2018
World Leaders in Research-Based User Experience. (n.d.). 10 usability heuristics for user interface design. Nielsen Norman Group. Retrieved November 4, 2022, from 10 Usability Heuristics for User Interface Design
Vendasta
Interested in working with us? Learn more about our open roles!
