When to Listen & When to Measure

Combining Qualitative and Quantitative Research

I get a lot of questions about when companies should talk to users and when they should ship code and learn from their metrics. Of course, the real answer is that you should do both constantly, but there are times when one is significantly more helpful than the other.

The most important thing to remember when choosing is this:

Quantitative research tells you WHAT your problem is. Qualitative research tells you WHY you have that problem.

Now, let’s look at what that means to you when you’re making product decisions.

Quantitative research tells you WHAT your problem is. Qualitative research tells you WHY you have that problem. -Tweet This

A One Variable Change

When you’re trying to decide between qualitative and quantitative research, you need to figure out how many variables you’re changing. One variable changes almost never require qualitative testing.

Here’s a simple example: You have a product page with a buy button on it. You want to see if the buy button performs better if it’s higher on the page without changing anything else. Which do you do? Qualitative or quantitative?

I don’t even know how you would test this qualitatively

That’s right, I said this one was simple. There’s absolutely no reason to qualitatively test this before shipping it. Just get this in front of real users and a/b test what percentage of them click on the button in the new location versus the old location.

The fact is, with a change this small, users in a testing session or discussion aren’t going to be able to give you any decent information. Hell, they probably won’t even notice the difference. Qualitative feedback here is not going to be worth the time and money it takes to set up interviews, talk to users, and analyze the data.

More importantly, since you are only changing one variable, if user behavior changes, you already have a really good idea why it changed. It changed because the CTA button was in a different place. There’s nothing mysterious about it.

There’s an exception! In a few cases, you are going to ship a change that seems incredibly simple, and you are going to see an enormous and surprising change in your metrics (either positive or negative).

If this happens, it’s worth running some quick observational tests where you watch people using the feature both before and after the change to see if anything weird is happening. For example, you may have introduced a bug, or you may have made it so that the button is no longer visible to certain users.

You should always test one-variable changes with quantitative metrics. -Tweet This

A Multi-Variable or Flow Change

Another typical design change involves adding an entirely new feature. New features almost always introduce or change multiple variables.

Here’s an example: You want to add a feature that allows people to connect with other users of your product. You’ll need to add several new pieces to your interface in order to allow users to do things like find people they know, find other interesting people they don’t know, manage their new connections, and get some value from the connections they’ve made.

This type of feature is made up of lots of changes to the interface

Now, you could simply build the feature, ship it, and test to see how it did, much the way you made your single variable change. The problem is that you’ll have no idea WHY it succeeded or failed — especially failed.

Let’s assume that you ship it and find that it hurts retention. You can assume that it was a bad feature choice, but often I find that people don’t use new features not because they hate the concept, but because the features are badly implemented.

The best way to deal with this is to prevent it from happening in the first place. When you’re making large, multi-variable changes, you’ll want to perform qualitative testing before you ever ship the product.

Specifically, the goal here is to do some standard usability testing with interactive prototypes, so that you can learn which bits are confusing (editor’s note: Trust me. There are confusing bits.) and fix them before they ever get coded and shipped to your users.

Sure, you’ll still do an a/b test once you’ve shipped the changes, but give that new feature the best possible chance to succeed by first making sure you’re not building something impossible to use.

Give new features a better chance for success with early prototype testing. -Tweet This

Deciding What To Build Next

Whatever you take from this next part, please do not assume that I’m telling you that you should ask your users exactly what they want and then build that. Nobody thinks that’s the right way to build products, and I’m tired of arguing about it with people who don’t understand how to do decent user research.

A much faster horse. SEE WHAT I DID THERE?

However, you can use both quantitative and qualitative research to help you decide what to build next.

Here’s an example: You have a flourishing social commerce product with lots of users doing lots of things, but you also have 15 million ideas for what you should build next. You need to narrow that down a bit.

The key here is to look at what your users are currently doing with your product and what they aren’t doing with it, using both quantitative and qualitative methods.

Use both quantitative and qualitative research to help you decide what to build next. -Tweet This

Qualitative Approaches:

• Watch retained users with your product on a regular basis. See where they struggle, where they seem disappointed, or where they complain that they can’t do what they want. Those will all give you ideas for iterating on current features or adding new ones.
• Talk to people who have stopped using your product. Find out what they thought they’d be getting when they started using it and why they stopped.
• Watch new users with your product and ask them what they expected from the first 15 minutes of use. If this doesn’t match what your product actually delivers, either fix the product or fix the first time user experience so that you’re setting expectations correctly.

Quantitative Approaches:

• Look at the features that are currently getting the most use by the highest value customers. Try to figure out if there’s a pattern there and then test other features that fit that pattern.
• Try a “fake door” test by adding a button or navigation element that represents the feature you’re thinking of adding, and then measure how many people click on it. Instead of implementing an entire system for making friends on your site, just add a button that allows people to Add a Friend, and then let them know that the feature isn’t quite ready yet while you tally up the percentage of people who are pressing the button. If nobody presses the button, don’t add the feature.

Still Don’t Know Which Approach to Take?

What if your change falls between the cracks here? For example, maybe you’re not making a single variable change, but it’s not a huge change either. Or maybe you’re making a pretty straightforward visual design or messaging change that will touch a lot of places in the product but that doesn’t affect the product flow too much.

As many rules as we try to make, there will still be judgement calls. The best strategy is to make sure that you’re always keeping track of your metrics and observing people using your product. That way, even if you don’t do exactly the right kind of research at exactly the right time, you’ll be much more likely to catch any problems before they hurt your business.

A version of this post originally appeared on my blog, Users Know.

Interested in more information like this? Take my class on UX Design for Growth on July 21st, 2015.

If you liked the post, you might also like the book! UX for Lean Startups.

Just like the blog, but made from trees.