Don’t Light a Fire In Your Tent: User Testing Without Burning The Place Down

Lighting the way

Mastering ways to produce light has been a pretty good thing for the human race. From the moment Prometheus stole fire from Zeus to give to humanity, we’ve been iterating new ways to use light to our advantage:

• Fire let us see and keep predators away at night
• Gas lamps let us navigate city streets on horseback
• Electric bulbs made it safe and pleasant to have light in our homes
• Batteries and flashlights made light more easily portable

These things all sound pretty good, and they are of course. But not all methods of producing light are good in all situations — you wouldn’t light a fire in your tent, would you?

Just like the universal acceptance of light as a good thing, user testing is almost a religion among Product Managers, Designers and Developers. We all seem to agree on one thing: getting feedback from actual users is good; not doing so is bad. But, just as there are many different kinds of light, there are different kinds of user testing, and they aren’t all appropriate for every situation. Some of them might even burn the tent down.

At FreshBooks, we’ve learned some lessons about user testing, like rising above excuses and just doing it, and some awesome approaches to integrating it into our regular Design DNA. It would be easy to declare victory at this point, and say that we’ve iterated our way to the best user testing methodology there is. The problem is, there isn’t one. There are a myriad of user testing approaches, and understanding the strengths and weaknesses of each one will help any organization on the path to getting the most out of user testing in different situations.

Let’s look at some examples.

Follow-Me-Homes:

“Follow-Me-Homes” (or site visits) are a powerful form of user testing done in your customers’ normal environment. While this might sound creepy (I highly recommend getting users’ permission in advance, rather than just showing up at their door), it can yield incredibly deep customer insights.

It’s sometimes known as a form of ethnographic field study by pointy-headed types (which is probably an accurate description of me on most days). Essentially, you recruit users who are willing to let you watch them use your product where they normally would (in their home, on the bus, at their office… you get the idea).

When it works: Follow-Me-Homes are great when you want to learn about your customers’ experiences with your product in their actual, everyday environment. This is no lab setting. When the single parent steps away from their computer every 5 minutes to deal with a crying youngster or answer the phone, you might learn something painful about the automatic log-out timing of your SaaS app that you wouldn’t have discovered by bringing customers into a lab or focus group.

When it doesn’t: Follow-Me-Homes require substantial time and energy to plan and execute, and are not particularly scalable as you need at least one actual team member to spend time with every participant. There is no “one- to-many” ratio here. It can even be destructive to the overall motivation for user testing in an organization, if there aren’t light-weight alternatives in the toolkit when user testing needs to happen quickly and frequently with larger numbers of customers. Follow-Me-Homes are best applied to generative research where you are looking for “surprises”, rather than for validation research where you have specific learning goals in mind.

In-office 1:1s or group usability testing

In-office usability testing generally takes the form of meeting customers 1:1 or in a group setting at your office, and showing them design mockups or prototypes.

When it works: It’s ALWAYS a good idea to get user feedback as early as possible to validate your design hypotheses, long before having a product or feature that is actually ready to go live. Sometimes, the most convenient way to do that is to bring customers to you. At FreshBooks, we’ve identified this as such a key part of our design process, that we bring in customers every single week. By making it part of our DNA, we don’t even have to think about it — it just happens — so there are no excuses not to get user feedback on early stage designs.

When it doesn’t: Don’t bring customers into your office to tell you about products or features that you already have in-market. If it has already “hit the streets”, then there are better and more efficient ways to get feedback about what you might want to change or add.

Clickstream analysis tools

If your product is a website, SaaS application or mobile app, there are plenty of well-known tools that allow you to measure user behaviour quantitatively. Products like Google Analytics and MixPanel are not always considered “user testing” tools, but used properly, they are very powerful ones. More widely known for tracking things like clicks and page views on websites, these tools can also be used to track actions and behaviours in SaaS and mobile apps. Imagine having a deep understanding of what processes in your app are resulting in the most customer log outs, or which pages customers spend the most time on, or what % of your customers successfully complete a task on the first try. Certainly sounds like user testing!

When it works: When you want to break down your user journey quantitatively into discreet steps or goals, and identify the drop-off points. When you want to validate the impact of a change you made against the success rate of completing a task, in a statistically significant way.

When it doesn’t: These tools are great for validation of hypotheses, but not-so-good for generative research. They show you quantitative user behavior, but tell you nothing of why these behaviors exist.

This is but a selection of the options in the user testing toolkit. There is a very comprehensive analysis of user testing methods written by Christian Rohrer.

It’s all good

In the end, we do all tend to agree that user testing is good, and not user testing is bad. I can’t think of a scenario where this doesn’t ring true. So if you have a new product idea, new feature, or want to change a form on your website, then by all means, DO user testing. There is no feeling like the excitement that a Product Manager, UX Designer or Developer feels when they uncover that new nugget of information from user testing that will lead to the next big growth driver or dramatically improve the customer experience. On the other hand, there is also no feeling like banging your head against a wall trying to learn something about your customers’ journey when you’re using the wrong method to get at that nugget.

Slowing down just enough to figure out which user testing method will actually get you the kind of learning that you’re looking for, will not only result in better learning, but should also keep your product teams engaged in the user testing process. So while light is good, please don’t light a fire in your tent.

Cameron Moore is a Director of Product Management @ FreshBooks