Defining User Testing and How We Use It for Our Product
User testing: what it is, how to do it, and our insight
At Bannersnack, we’re trying to make our product as polished and as user-centered as possible, so in order to carry out this important task, we have been relying more and more on user research.
What is User Testing
User testing is a qualitative research method, part of user research, that implies testing a product, feature, or a prototype with real users.
The main objective of user testing is to find and identify possible problems that can occur before the release of a project or to detect and correct the ones that are in an ongoing project.
“The problem is there are no simple right answers for most Web design questions (at least not for the important ones). What works is a good, integrated design that fills a need — carefully thought out, well-executed, and tested.”
Steve Krug
The insight that a user testing can give is tremendous, not only because we get to see how our customers or possible customers use our platform, but also to validate or determine if our proposed design works as intended.
Finding the users to test with
Recruiting users for usability testing can be a challenging task; finding people that fit our product persona requires a bit of researching. But we found that a good way to quickly test our product was to ask new employees — who haven’t yet interacted with our product — to come to test it out during their onboarding week. Our all-in-one visual production platform is addressed to marketers and designers, which is very fitting since most of our new hires are marketers or designers, and even those who aren’t marketers and designers still offer valuable insight.
Besides Bannersnack’s new hires, we can often find a user testing group in places where people are grouped together. For example, universities, forums, IT/tech groups, and Facebook pages/groups that are relevant to our platform’s area of interest are all great sources of finding users for testing.
When you have multiple candidates to choose from, they need to be filtered by availability and qualification. Gathering information about them can be done via an online form, e-mail, or by telephone.
Preparation prior testing session
Besides finding users that fit your testing needs, it’s also important to take the testing space into account, as well as preparing the tasks that the user would have to attempt.
Choosing the tasks and creating scenarios
You can choose to give users between five and ten tasks per session, depending on how complicated and how long each task takes to complete.
Your decision on which tasks need to be tested can be a mix of the following types:
- the critical tasks
- the tasks that have information that is not very clear to the user
- the tasks that you have doubts the user will do successfully
After tasks are selected, they need to be included in scenarios that offer more context and they need additional information to give users a direction.
When scenarios are finalized, a good practice is to print them, and cut each one into separate pieces, so as the user does the tasks, we will give them the respective scenario. Having them printed helps the user refer back to them, should the case occur.
Resources and Equipment
One of the first things to keep in mind for user testing is a quiet room that is not very transited by people; you don’t want your user to be distracted by outside factors during testing. Their focus has to be on the test in order to get the most accurate insights.
At Bannersnack, we use a large TV that streams what the user does on the laptop because this allows others to see what the user does from a distance, without them feeling stressed about having a bunch of people right behind them, watching over their shoulder. With the user’s consent, we use the Loom app to record everything they do on the platform, along with the conversation we have with them.
People who conduct the test
We choose one facilitator that serves two roles:
- To guide the user by letting them know what they have to do and answer their questions
- To take on the role of a therapist, who has to be able to see the user’s interaction and get them to verbalize what they are thinking and trying to do
The facilitator must stay neutral, being careful not to influence or help the users in solving the tasks they need to do.
Along with the facilitator, there are one or more observers that take notes on what the user does and what problems are coming up.
What methods we use for testing: testing prototypes and the real product
When it comes to choosing the methods for user testing, we have two options, depending on whether it’s a prototype or an actual product.
1. Prototype testing
In the case of prototypes, we searched for options to test the designs we made in Sketch within a platform that we can send them to a specific user base (for testing).
There are many platforms for testing prototypes: UsabilityHub and UserTesting, but in our case, what works well for what we need is the maze.design platform. It has an Invision integration, incorporating all the transitions and effects from it.
Maze.design offers really helpful reports with how users tested our prototypes, with detailed insights including misclick rate, average duration, average success, heatmaps per screen, and more.
Our process regarding prototype testing is to upload our designs from Sketch to Invision and create an animated prototype there. Sketch cloud can also work with maze.design directly, but we discovered that the downfall was that it didn’t have hover animations. That being said, we decided to go with Invision for prototyping just because they have that feature. We add the share link from Invision with the prototype to maze.design, where, after a few settings, our prototype is ready to be shared.
The link from maze.design with the prototype testing can be sent either to a group of users from our database, or internally to coworkers from which you can get useful feedback, especially if it’s a big company.
Take note that this method of testing resorts more to quantitative data rather than qualitative, offering an indirect assessment of the usability, measuring users’ performance with a specific task.
2. Live product testing
When it comes to testing a live product, we rely on user testing with participants that we can observe. We haven’t done any remote user testing yet because we prefer to test them in real life, but we are planning on trying our remote testing in the future as well.
After all the necessary elements for running a successful user testing are ready to go, the testing can begin. Before starting a test, however, you have to select a few tasks for the user to do and write scenarios for each task.
One of the tasks we often give to users when testing is to replicate a relatively simple design that we show them before. By doing this, we can observe all the significant steps a new user takes on our platform and observe the process of visual creation.
After that, we give them specific tasks that cover the main areas the user interacts with, to see if they come across anything confusing or misleading. When the tasks are completed, if there is something we think we need more insight on, we return to that specific aspect, making sure we clearly understand what and why they chose a particular way of doing things.
Users often think that the test is about them and they get stressed out. That’s why it’s important to mention that we are not testing them, but rather our application, to see any design-driven problems that we didn’t notice. They essentially need to know that they are not held responsible if they don’t perform well during the session.
We usually run testing sessions that last 1 hour and a half up to 2 hours, depending on the time available. Often, the recommendation for a user session is one hour, especially if there are multiple sessions in a day. Still, taking into consideration that we focus on our users to create a visual and that we usually have 1–2 sessions in a day, we exceed that one-hour recommendation, to do things at a steady pace.
The debriefing
The debriefing must contain a list of usability problems that resulted in the user testing sessions. After that, they will be sorted for importance and prioritization by their seriousness and by their frequency.
Based on the notes that are made by the observer during the testing session, we begin to discuss and highlight the main problems that occurred, and differentiate the usability issues with different colors according to how easy it was for the user to complete specific tasks.
All the gathered information should be included in a report that contains the following:
- the user’s job qualification/line of work/expertise
- testing date
- people involved in conducting the testing session
- the recorded video
- the task description
- session notes: user’s actions and details regarding the usability issues they encounter
Using the data we collect
After just five sessions of testing, we can already identify trends. The more the product will be tested, the more the feedback will start to repeat itself, and the new insights will be less and less apparent. By gathering the reports from each one and analyzing them, we can see the usability problems that are common and how serious they are.
The next step is to prioritize them by importance and urgency to be solved and split them into tasks that need to be addressed. After new solutions and iterations are made, they need to be evaluated and tested again, and if the proposed solutions are working, the improvements or new features will go live.
Things we learned from user testing
Combining data from user testing with the ones coming from support and other platforms (Hotjar, Intercom, etc.), we get powerful insights on how we can improve and improve our product so that our customers have a better experience.
User testing is an essential part of creating and developing a product, both in its early stages and after it’s released. There are issues that even the best UXers might not catch, and that only come to surface after user testing sessions.
Also, if there are debates between designers regarding which of the solutions will work better, user testing can provide an unbiased response.
Takeaways
- Start testing as early as possible — not during the finishing steps of a product.
- User testing is part of user research that can reveal how people use your product and where they meet difficulties along the way, giving you great insight on how to improve the product.
- Prepare a quiet room that has a large screen and a station where the user can work.
- You will need a facilitator and an observer to take the test with one individual.
- Debrief as soon as possible after the testing is finished, by making a list with the encountered problems and sorting them according to their importance.
- You can recruit people for user testing by reaching out to universities, Facebook groups/pages related to your needs, and even friends. Also, don’t forget to test your product with the new hires in your company.
Recommended resources:
