Image from Veri Ivanova: https://unsplash.com/photos/p3Pj7jOYvnM

Optimizing Less than Ideal Usability Testing

The benefits of usability testing are undeniable. You get to test your ideas on the fly and pivot relatively fast toward better and more robust solutions. However, it isn’t always possible to do thorough usability testing. Either there are time constraints — like you have to do big iterations within a short period of time, or there are environmental constraints — like your users cannot allow you to screen record them. Don’t let constraints deter usability testing. They just mean that you have to get creative to get the most bang for your buck.

Luckily, I’ve had a number of years of experience conducting usability testing in less than conducive environments. In doing so I’ve developed more than a handful of tricks to help the testing be as successful as possible: tricks that I’m going to share with you.

Before I get started, I’d like to cover the basic constraints. There are three basic types of resources time, money and rigor.

Time — If you are short on time, you are going to pay for it by using more money to afford more people or you are going to pay for it by of only capturing the larger usability problems.

Money — If you are short on money, you are going to pay for it with time because the testing is going to take longer or the testers are going to lack the ability to properly document all of the usability bugs.

Rigor — If you are short on rigor, you are going to make up for it in terms of time by having to do repeated testing AND in money because you are going to have to pay for all that additional testing.

So, if you know you are going to be short on one of those three things, plan accordingly ahead of time so that you can be the most productive. The more you know your constraints the more successful you’ll be.

The last thing I’m going to say before I get to my tips is that for my work I’m fairly restricted. For this reason, I can’t take pictures, I can’t video record, and I can’t screen capture. The tips provided below help me deal with that situation.

Laurian’s Usability Testing Tips

1. Friends don’t let friends usability test alone. This is my most basic tip for a handful of reasons. The first is that usability testing is very stressful. You *need* someone who is going to start calm and maintain a calm presence when interacting with your users. So one person needs to be in charge of direct eye contact and thinking super calm thoughts; the other person needs to be writing down notes as fast as humanly possible. The note writer gets to be frantic and look squirrelly — it is the only benefit of being the note taker. Sometimes it helps to switch these roles back and forth between testers, but it helps everyone if you know what your role is. Don’t head into testing without deciding who is going to do what. Also, there are just some strange usability tests where the participant goes off the rails. Either the participant won’t stay on task or their behavior is less than professional. Having two people there is brilliant because the other person can step in and rectify the situation — like good cop and bad cop. The last reason is verification. If you aren’t sure you saw a usability problem you can always run it by the other person to make sure.
2. Plan like it is the day before the first day back at school. Do you have all your materials printed and ready the day before? Good, because printers are the worst and *always* run out of toner right before a usability test. It is like the printer can smell the user coming. Did you double check all of your spelling and grammar? Did you cut out all of your usability tasks and paperclip them in order? Did you staple all your packs of surveys together and then collate them per participant? Good. Now you are ready.
3. Dry runs are for winners. You know what most people forget? Usability testing the usability test. I’m not even kidding. Grab a friend, or heck, grab Frank in the next cube (hi Frank!), and have that person do your usability test. The less the fake participant knows about the product the better. Doing a dry run of your test is going to show you the weaknesses in your methods early on. For instance, this is where you’ll find out if your task wording is confusing or if you forgot a step in your protocol.
4. Electronics fail when you need them the most. OK, so I told you printers are the worst. But that was a lie. You know what is the worst? Electronics. Pretty much anything electronic. If you are relying on your participants filling out surveys before or after the study through an electronic survey, guess again. It is like there is some god out there that just likes to mess with usability engineering. I suggest printing out at least one survey for every potential participant as a backup and carrying it along with you. In fact, this has happened to me so many times that I no longer rely on electronic surveys and I always provide paper. It is just easier and I think there is something calming about writing on paper.
5. Always pack spare pencils. This one seems silly, but it is a staple. No matter how many pencils you think you need, you’ll probably need more. I carry a little bag of pencils with me when I go to tests.
6. Screenshots — take them of everything. My favorite thing to do right now is to take a screenshot of every potential screen that the user can see. I try to stack them in the order that the user will do the task. I then print them out at about 2 images per page and use them to help me take notes. As the user moves through the test, I’ve developed a little short hand. I write ‘x over wrong clicks, ‘? for where the user expressed confusion, and circles for when the user moved the mouse over the target area but didn’t click. As the user does their task, these printouts help me document most of the usability errors I see without spending a ton of time writing.
7. Create a script; know the script. Usability testing is a bit like public speaking. You need a decent amount of practice but after the practice the whole process isn’t as scary. One of the tools you can use to help you be prepared is to write what you want to say for every stage of the test. I call this writing the script. Most people just write the protocol (which means what steps and when to do them). I like to take this a step further. It helps ensure that the same things are being said from participant to participant and it helps solidify how to get there. That said, if you are short on time, make sure to at least write a generic script so that you cover consent, thinking aloud, and the purpose of the test.
8. Label all outputs with a unique signifier for each participant; decide these ahead of time. I’ve had cases where the surveys get mixed up. Since then I’ve learned to write down the participant number on every paper before handing them to the participant. This ensures that items do not get mixed up when you have a big stack.
9. Don’t worry about associating errors with tasks. This is a little controversial, but I’ve stopped worrying about whether I got all of the errors associated with the right task. Instead I give tasks three possible scores: Completed Successfully: 2, Completed with Difficulty: 1, Incomplete: 0. Rating task completion in these rough categories frees me up to focus on documenting all the usability errors without worrying about relating them to specific tasks. This means I can write down more and stay more focused rather than tracking tasks.
10. Write up your notes as soon as possible. Soon all of the tests will start to blur together in your head. Make sure to write your notes in a well formatted way as soon as possible. One of my colleagues is very reluctant to do back-to-back usability testing because of participant blurring. She is on the mark, but sometimes you don’t always have that luxury. Definitely don’t do more than three participants without time for documentation, and allow yourself at least 2 hours for each participant’s notes.
11. No more than 10 tasks. If you have more than ten tasks, unless your application is very simple, you’re participant is going to burn out. This is especially true if you are sitting there watching your participant at their side. It is mentally stressful for someone to sit beside you and watch your every move. Make sure your tasks are concise, direct, and free of fluffy text. No “‘pleases and thank yous” in your tasks. Keep those verbal.
12. Team cohesion is your strongest asset. The last tip I want to provide is about team dynamics. I have the luxury of working in a team of some of the best UX people. We’ve also worked together for a while. There is much to be said for the value of good team dynamics that adds unquantifiable value to a usability test. Whether it is the ability to pick up where the other left off, or just knowing which of you can handle particular participant personalities, usability tests can be difficult. Making sure that you work well with your partner is going to ensure a better set of results. This means that if you haven’t met your partner make sure to do a dry run together. Ask about your partner’s weekend. Because, if you can’t develop a rapport with the person you’re testing with you aren’t going to be able to establish one with your participants.

Here is another one from my team that I agree with:

Users aren’t perfect. When we think of our users we account for different user groups, but we don’t necessarily account for the fact that all of our users aren’t going to work optimally. Users are going to have different skill levels and understanding of technology. When doing usability testing it is great to remind ourselves (and even our users) that people are going to do tasks at different paces. Being patient, especially when you are testing with time-boxed tasks, is going to make everyone happier.

Those are our best tips. If you can’t tell, I leave little to chance and write everything down. I would recommend the same for you. What pro tips do you have to share for doing usability testing on the run?

If you are interested in learning more about Next Century, please visit our website below:

Next Century