What I Learned from an Afternoon of Guerrilla Usability Testing Treehouse.com
Part One: A how-to for quickly and easily understanding the pain points of your product
I am a huge fan of online learning. When building my career as a photojournalist, I learned everything I needed to know by combining action with online study. I did the same when I went on a rap tour as a videographer, without ever having made a single video. Caution: this is a rap video, and thus contains copious amounts of inappropriate behavior.
When I became obsessed with design, I immediately started consuming all the online information I could. I quickly realized that if I wanted to work with my engineer comrades, I should learn some HTML and CSS so that I could at least speak some of their language. I took the Team Treehouse online course “How to Make a Website” and was blown away by the efficiency of the lessons. I immediately loved their product and decided it would be fun to try out some Guerrilla Usability Tests to see where improvements could be made.
Note: I am in no way affiliated with Team Treehouse. This was a personal project I did to further my learning.
Why Usability Tests?
Usability tests are an amazing way to understand users’ pain points with a product. When a team is working all day, every day on a project, they naturally build a long list of assumptions regarding what users will immediately understand. Usability tests put these assumptions to the test, and imbue the team with empathy regarding the frustrations their users are experiencing. Often, teams neglect to do enough usability testing (should be happening every week if the team is iterating quickly and efficiently), because of the time and energy required to set up and run these tests. If your team falls into this boat, guerrilla usability tests are a great way to dip your feet in the water.
When to Use a Guerrilla Usability Test
If you have time and resources, a Guerrilla Usability Test is not for you. In a perfect world, your team would recruit participants that fit your market and align with your persona. With some products, say a B2B accounting software that is targeting power users, a Guerrilla Usability Test on coffeeshop laymen will be virtually useless. With others, especially those with consumer-facing products, it can provide an astounding amount of value given that it only takes an afternoon to complete.
How To Do It
- Decide which aspect of your product you would like to explore.
- Devise a task that requires users to engage with that aspect of your product and has a clear goal.
- Get a screen recording tool: I’m a huge fan of Silverback.
- Write a script.
Your script should include:
A) an explanation of who you are
B) a reward for their time
C) an ask for permission to use the screen-capture software
D) an explanation that they can do no wrong — all you want is for them to talk you through their thought process
E) an explanation of how your goal is to see people’s process without help, so you won’t be able to answer any questions during the process!
F) a clear and explicit task for them to try and accomplish
You can see an example of the script I used here:
Hey, I’m a web designer and I’m looking for people to try out a website and tell me what they think. It’ll take about 20 min and I’ll give you a 10 dollar gift card as a thank you.
No? Bummer, Thanks anyway!
Yes?
Okay I’m going to bring you to an education website called treehouse.com. All you need to do is find a course you might be interested in taking, complete the first lesson, and tell me what you think. I’m going to use screen capture software to record your reaction and how you move through the site so that I can compile data on where people have trouble later. Is that okay with you?
There are two keys to making this super helpful for me: the first is that you do your best to just constantly talk me through your thought process. The more I can hear why you’re doing what you’re doing or what your confusions are, the better! The second is that I want to understand how people interact with the site — not with some guy walking them through it, but how they really would if they wanted to check it out themselves. So if you get stuck or have questions, I won’t be able to step in to tell you how to do something. Just do your best! We want to see where people have trouble so we can redesign it to be easier to use.
So pretend you heard about treehouse.com from a friend and learned that it’s a website that has courses on coding, web design, and other tech-related stuff. You are curious about learning about some of that stuff so you wanted to check it out. Your task is to find a course you’d be interested in taking and complete the first lesson.
Ready?
Once you got your script, approach strangers as non-creepily as possible
I’ve found that coffeeshops with quiet back patios are best. Memorize your script so you can be comfortable and accurate in your explanation. Take a deep breath and go for it.
Do 5 tests.
According to the legendary Nielson Norman Group, the amazing Daniel Burka, and the guy who wrote the book on usability testing, 5 is the optimal number of people to test. Learn what you need to know and get out of there!
Bust out your stickies
Now that you have 5 tests recorded and in the books, it’s time to go through your recordings. Get out your stickies and write down every relevant point a participant makes about your product. I’ve found it’s best to record these in quote form for three reasons: 1) it’s super fast and easy to just write down what people say as your watching, 2) by framing these pain points in terms of things actual people said, it furthers your connection with the people who use what your building, and 3) when working on a team, it can be helpful to talk over exactly what people may have meant when they made a point. If you synthesize what they say on your own, you make include your assumptions in your synthesis.
Get Sorting
After you collect all your nuggets of wisdom, it’s time to head to a whiteboard or a wall and start grouping them into categories so you can see which themes emerge.
In my test on Treehouse, it looked like this:
Now that you have your pain points grouped into categories, synthesize them down into core pain points. I recommend choosing one quote that illustrates each core concept.
For me, the main issues broke down like so:
Pain Points
A) Inability to easily jump around to answer questions as needed.
“Having the ability to work on something in the context of your job is way more powerful and useful” -Jordan
B) Inability to easily use the course in conjunction with a personal project.
“You prove the skill by building something” -Aline
C) Level of difficulty wasn’t quite right (either too easy or too complex).
“I hate being told how to download something from the app store” -Aaron
D) Lack of ability to just start coding and learn through action.
“I learn as I do it” -Jordan
E) Having to watch lengthy videos.
“I’m gonna skip ahead to where I can actually do stuff.” -Alex
Current Strengths
A) Clear call to action for beginning course
“Well that’s simple enough, going to Start Course” -Alex
B) Getting points and positive feedback for completing a lesson.
“Woooo! 12 points!” -Sam
I find it valuable to include the positive feedback for two reasons. One, sometimes doubling down on a strength is the correct design decision, and two, it’s crucial to be aware of what’s working so that you can avoid disrupting those things if you do decide to focus on solving the pain points.
Understanding the problem is only half the battle
To see my design suggestion based on these tests, take a look at Part Two of my Treehouse Redesign series.
Written by Toby Silverman
As a former classical ballet dancer, kindergarten teacher in Seoul, rap tour camera man, and VICE photojournalist, I’m a 26 year old Product Designer with a unique perspective.
I’m excited about the intersection of technology, art, and design.
I live in San Francisco.
Check out my work at design.tobysilverman.com
