A Product Manager’s Guide To Usability Testing
A Cup Of Tea
You came home after a long day and are ready to have some relaxing tea. You placed a tea bag in your favorite cup, boiled water, and poured it into your cup. You are all set! Mmmm it tastes good.
You performed all these tasks without even thinking about the products you used, right? The cup, the tea bag, and the boiling pot. All of them are designed to enable you to reach your goal: making a cup of tea for you to drink. You did not think whether you were able to reach your goal, you just did them. ( I mean who thinks about that anyway…).
What if the pot you have looked like this:
Noticed anything? Yes? No?
Imagine yourself holding that pot and trying to pour the water to your cup.
Noticed now? (I can hear you saying, Duh!).
You cannot easily pour your water, right? It means that the pot is not easy to use, and you are not able to achieve your goal. Whoever designed that pot did not do a very good job, but you are only realizing whether this was a good or bad design because it bothered you. There is a reason for that.
According to Don A. Norman — founder of Nielsen Norman Group and the author of many famous design books including Design of Everyday Things:
Good design is actually a lot harder to notice than poor design, in part because good designs fit our needs so well that the design is invisible, serving us without drawing attention to itself. Bad design, on the other hand, screams out its inadequacies, making itself very noticeable.
One of your responsibilities as a Product Manager to provide a delightful experience to your users. The first step to achieve this is to ensure your users are able to perform the tasks that they want (and you wish them to) easily even when they use your product for the first time. You want your users to be able to boil and pour the water to a cup without thinking. So how can you understand if your product is able to achieve this? By conducting Usability Tests regularly.
If your company already has UX Researchers that conduct usability tests as part of their role, you should brag about this when speaking with your fellow PMs :) because you can work with them when they perform these tests and integrate the significant results into your product requirements to enhance your user experience.
If your company does not have UX Researches, then you and your Product Designer(s) need to work together to conduct these tests.
When To Conduct Usability Tests
You should conduct usability tests when you are trying to understand the difference between what you think your users will do with your product and what they actually do/can do… which is always.
You should treat Usability Testing as a continuous process rather than a point-in-time activity. Your aim as a Product Manager is to continuously learn and iterate on your product before and after launch.
If you are conducting these tests before launch, you should do it before any development work and use prototypes (low fidelity wireframes first and then high-fidelity final designs).
How To Conduct Usability Tests
- Create a Test Plan
- Test
- Analyze Your Findings
- Communicate
Create a Test Plan
If you are a person who believes in living spontaneously, then can you skip this step? In the end, it is not about the destination it is about the journey, right?
No, of course not. It is about the destination in this case, so no adventures for us. You should start by creating your test plan. At this stage, you will answer the following questions :
Who will you test with? You need to identify your testers. It is important to test your prototypes with people who are representative of your persona/audience. However, if it is hard for you to reach your potential users then you can test with anyone! Anyone?? Yes. Studies show that many of the most serious usability problems can be encountered by almost anybody that uses the product.
How many people you will test with? The simple answer is 5–10. According to Nielsen Norman Group, the first five users that you will test with are very likely to encounter many of the most significant problems related to the tasks that you will test. Testing costs increase with each additional study participant, yet the number of findings quickly reaches the point of diminishing returns.
Will you compensate them for their time? If yes, how?
How will you find them? Internal employees? External People (Guerilla)?
How will you conduct your tests? In-person? Remote? If you will conduct your tests remotely, what tools will you use? One quick note about remote user testing: Make sure to record these tests so that you can re-watch them later again not to miss any details. Few fancy tools that can help you conduct your remote testing: UserBrain, UserTesting, Userlytics.
Consent Form: If you will conduct your user tests with external people then you will probably need to write a consent form for them to sign. The consent form should explain what the study is about, what should they expect, how will you use their data, how long will you keep their data, risks, benefits, etc. You may need to work with your legal team to make sure form covers everything, or your company may already have a standard format.
Which tasks will you test? The big question! This is the most important part of your test. You should work with your Product Designers to identify the tasks that you aim to test. Writing these tasks efficiently is another skill to master which I won’t explain here (Maybe in another article!).
Example task 1: If you were to purchase a yellow t-shirt using our product, how would you do it? (E-commerce website)
Example task 2: You want to rent a car to go to LA from SF, how would you do it? (Car Rental App)
What will you say to your users? You should write a script beforehand that includes the purpose of your study as well as the tasks that you will ask them to perform. You should also use the same script for each participant so that your tests are consistent and unbiased.
What are your considerations? What areas will you test? What are your success criteria? Common metrics you can use: Time to complete/task, Completion Rate, Error Free Rate, Expectation Matching (there are many ways to go from point A to B. Do they perform the steps you expected them to take to reach from A to B?).
Assign The Roles: Can you ask questions, listen to answers, watch your user, take notes along with your comments, and smile at the same time without any delay? If you can that is awesome! You should write a Medium article about it and teach us how! Most of us cannot. That’s why it is always recommended to have at least two people when conducting user tests. This could be your designer and yourself. One of you can ask the questions (whoever likes to talk more) and observe your users while the other one takes notes.
Phew! That was long! But we are finally ready to test.
Test
At this stage you… test.
But of course, I have a few tips:
Tip # 1: Make sure you treat your participants as regular people rather than participants :) You can start by not using the word participant. What I realized is that when people think that they are part of a study they get too nervous and don’t act like their normal selves. Have a warm introduction about yourself and your teammate. Maybe add some dad jokes. They may not laugh but they will at least know you tried🙊.
Tip 2 #: Don’t just listen to their answers to your questions, observe them. People say one thing and do another (Surprising huh?).
Tip 3 #: Make sure to let them know that you are not testing them but rather you are testing your product. Generally, participants feel like they should do everything right and answer everything right. However, of course, they don’t know that there are no right answers. Make sure they know that there is nothing that they can do wrong.
Analyze Your Findings
After you are done with your interviews you and your teammate should come up with two lists: list of the main problems you identified and the list of the problems that you will fix. The main problems can be identified based on the frequency of occurrence and severity. After you come up with your initial list you need to prioritize which ones you will fix first. You can use a common prioritization technique here: Cost-Benefit Analysis (CBA) aka Effort versus Impact.
Communicate
After you come up with major problems and design solutions to them by working with your designer(s), you need to share your results with your stakeholders. Especially with your engineers who will build these features. Make sure they know the why behind these decisions. In this case, you can go over your CBA so that they can understand your thought process.
Final Thoughts
It is important not to forget that every organization is different and you may never have to actually conduct these tests by yourself. Sometimes, Product Managers just attend to these tests as an outside observer while UX Researchers and Product Designers conduct them. However, I always believe in learning everything that will benefit my product if and when I can.
