Simple and Continuous User Testing
As product managers, we firmly believe that success is mainly a byproduct of great products. And as much as we like to be data-informed, we still recognize the limitations of a purely quantitative approach: When looking at quantitative data, you learn a lot about how users behave, but it is close to impossible to determine why they behave in a certain way. To gain these kinds of qualitative insights, user testing is the way to go.
Performing in-house user tests involves less effort than you might expect and the benefits are huge when it comes to finding UX bugs. Analytics tools may show you where users struggle when using your product, but user tests reveal why they struggle. It’s as simple as observing users using your product, performing tasks you ask them to do, and talking to them afterwards. You will see where they encounter problems and you’ll get the chance to ask why they had difficulties at some point.
I have already advocated the importance of qualitative user feedback in a motivational letter — The points made there still hold true today. So here is a brief how-to on establishing continuous user testing in your company.
Keep It Simple!
Performing user tests, there can be a tendency to overdo things. From my experience, this is mostly unnecessary. User testings don’t need to be big events. Usually, you just need a website or a prototype, a set of tasks and questions, and, of course, some test persons. You can then easily record a screencast, webcam video, and audio track of each test session using Quicktime — no need for additional tools. A partner who can take notes while you’re supervising test sessions comes in handy when you want quick summaries.
Real Users, Regularly
The most important thing is to perform tests with real users on a regular basis. From our experience, a biweekly cycle makes sense. Take these regular dates seriously and don’t find excuses like “The product isn’t ready yet” or “Next week I’ll have more time”. As soon as your user tests become routine, you’ll recognize how valuable these qualitative insights are. This will make it easier to place importance on keeping your appointments.
It’s also advisable not to invite more than four test subjects per day. This will prevent you from spending excessive time and effort on testing. You should be able to have a more or less normal work day and to consider your test sessions as ordinary meetings.
How to Find and Invite Users
Inviting existing customers is fairly straightforward. If possible, simply invite a number of customers who live within a 30-minutes radius of your office. Somewhat surprisingly, people are often happy to have a look at the office and meet the team behind the products they use. After some experimentation, we found that customers are most likely to show up when they receive the invitation eight days in advance with a reminder a couple of days before the testing.
Here are some tips for the invitation email:
- Address your customers personally. Since you are not going to be inviting hundreds of test subjects, this doesn’t require a lot of additional effort and significantly increases the likelihood that customers will accept your invitation.
- Do not focus too much on aligning the text with marketing. It should be a personal invitation, not an advertising email.
- In our experience, offering customers a compensation for their participation did not increase the response rate. Sometimes we mentioned of a “small thank-you” but left open what it would be in order not to raise expectations too high — in any case, you want participants who are willing to help and not the ones who merely want free gifts.
Your Device, Their Device
In general, test persons don’t bring their own personal computers to test sessions. And even if they would, we usually want to avoid the hassle of using password-protected staging versions on private computers or connecting external devices to our internal WiFi. However, testing on an unfamiliar device proves to be difficult for many users. Chances are that they will struggle with an unfamiliar keyboard layout, input device, screen sensitivity, or the like. Ideally, you should offer them a selection of devices so that they can pick the one they are most familiar with.
In addition, if you already have a live version of your product, let your users test it on their own cell phones at the end of the test session. Most probably, you will learn about some issues you weren’t aware of.
How to Learn the Most
Before actually testing your product or prototype, let your test subjects start with an empty browser and observe how they navigate, which keywords they use, etc. You’ll learn a lot about your competition in the process. In general, emphasize that you’re not testing the user but the user is testing your product. Apart from that, avoid guiding your test subjects too much since this will corrupt the validity of your test. If they struggle at some point, encourage them to go on and try harder. Observe their behavior, facial expressions, and eyes. Only discuss encountered issues subsequent to each task in order to not distract the user. On the other hand, if you identify obvious errors during a session, fix your prototype immediately if possible. You’re done when five users in a row had no major issues using your product.
When your subjects have completed all tasks, you can seize the oportunity and ask how likely it is that they would recommend your product to a friend. This way, you can determine a rough net promoter score for your current version. Also discuss additional questions regarding your competition after the session and ask your test subjects for their opinion on your product as well as those of your competitors.
Avoid Confusion
Testing a product, users tend to enter their real data, which is good in terms of realistic user behavior but may lead to confusion. For example, a staging system may have stub payment and logistics providers but be connected to a live email provider. Participants may feel tricked when they receive order and shipping confirmation emails after placing a test order. So make sure to clarify these kinds of things beforehand.
Power Users with Power Opinions
Power users tend to have strong opinions on your product and the features you should implement. Sometimes their ideas are very useful or even already on your roadmap. In other cases, their requests are overly specific and not beneficial when accounting for your customer group as a whole. It’s not always easy to determine whether a power user’s idea is of general interest, so weigh their opinions with care.
Lessons Learned
- Keep it short and simple!
- Have different devices prepared to account for your customers’ preferences.
- Send out invitations about eight days in advance.
- Sending a reminder one or two days before the test date is good practice. This contributed to keeping our no-show rate below 25%. If you can, call people the day before, this will reduce the no-show rate even further.
- In general, customers are happy when their feedback is appreciated, but it’s nevertheless good to have a small thank-you at hand.
- Base your prioritization on what you derive from the observed customer behavior and not on what they say they need.
- In general, user testing is more beneficial in terms of UX bug hunting than for prioritizing new features.
- You are done when five users in a row like your product and don’t encounter any UX hurdles.
This post was written in collaboration with Niels-Ole Kühl.
