The Value of Asking Users Qualitative Questions As A Product Manager

Paul Lopushinsky
Jul 20, 2017 · 5 min read

Asking the right questions is one of the most important aspects of being a Product Manager.

It’s right up there with empathy as arguably the most important skill to have as a Product Manager.

Ask the wrong kinds of questions, and you’ll gain misguided insights and make decisions that don’t align with what your users actually want. To quote one of my math teachers from high school:

You’re on the right track…but the wrong train!

There are many kinds of questions that you can ask that can potentially lead you in taking the wrong train.

First thing that popped up in Google images typing “wrong train” Source: Annie’s Simple Life

When you use questions that are leading, biased, or loaded, questions with yes or no answers, or 1–5 scale answers, you may get answers that you want, but not the ones you need. You are also missing out on a goldmine of insight.

To clarify, there is still plenty of value out of quantitative questions.

You may have hundreds, thousands, or millions of responses to these questions, and it’s very easy to quantify them to gain insights. However, you can’t just rely on these questions to gain insight into how users are using your product.

Qualitative questions requires you to sit down and go through them. That said, taking time to sit down and read these answers can give you incredible insights on how people view and use your product.

I want to discuss a qualitative question that I use to gain high levels of insights from users.

Here is the qualitative question I like to ask:

What are some of your biggest challenges or pain points with using X?

That’s it. Nothing fancy. Here are some examples of using this question.

  • I was curious in regards to improv what some of the biggest challenges that people face are outside of improv. So I asked the following: “Outside of the improv portion, what are some of your biggest challenges with improv?” I gained very interesting insight that I wasn’t expecting — by far the biggest challenge people had with improv was marketing their shows!
  • If you want to validate an idea for a product (say you want to produce a Javascript learning product) you could ask “What are some of your biggest challenges in regards to learning Javascript?” You’ll gain insight into potential customer pain points that they have, and you can better produce a product that addresses these problems.

What makes the “biggest challenges” question so effective?

  1. It’s an open-ended question where you’ll receive open ended answers.
    You get to the heart of the matter, and you find out what are some of the biggest pain points that your customers are having.
  2. You get what you ask for, which is your customers biggest pain points.
    It’s easy to ask questions that will give you answers that you were hoping to get (a biased/leading question). With the biggest challenges/pain points question, you get what you ask for, and may get answers you didn’t want to hear, but needed to. Perhaps you’ll learn that a lot of people think your product sucks and are forced to use it at their work because it was mandated by the higher-ups. Ouch, talk about a wakeup call!

As mentioned, it takes more time to go through these answers than quantitative questions, but the value from these answers is far greater than quantitative questions.

Customers love to talk about their challenges, their concerns, their pain points, and if you give them the opportunity to, they will.

If someone sends you a long survey full of 20 disagree-agree range questions to get insight on their product, what’s your first reaction? Here’s mine…

Into the trash it goes! Unsubscribe!

To clarify once again, there is nothing wrong with quantitative questions. You should be asking them!

They are very valuable. You can check overtime how customers opinions are changing. Perhaps surveys have indicated more people are likely to recommend your product compared to a year ago. However, it’s easy to get into these points of data that can simply be ego stroking, and ignore the heart of the matter and finding the pain point your customers have. You need to supplement them with qualitative questions as well.

To repeat, you must to be asking both qualitative and quantitative questions.

To also note, you’re opening the floodgates for people to tell you what sucks about your product. How comfortable do you feel with that?

Asking the “biggest challenges” question makes you very vulnerable. You’ve asked your customers to identify the biggest struggles that they face with your product.

So, what do most people and companies do? They avoid asking these kind of qualitative questions. They fall back on quantitative questions and metrics that are easy to measure.

I’ve worked with companies or clients in the past that were reluctant to ask this kind of question, but they were glad when they did.

One client was far too focused on things like NPS and avoided talking to their customer, and once they reached out to them with an open ended question, they began to see just how off key they were with what their customers wanted, and were able to course correct with these insights they gained from asking open ended questions.

In my own experiences, I’m currently validating a product idea with a friend, and what we initially had in mind was far, FAR different than how it currently looks. We asked the biggest challenges question, and what we thought was going to be a major pain point for customers was barely mentioned, while another factor was mentioned in about 95% of the responses we collected. Needless to say, shifted focus in the product that we are seeking validation on.

Conclusion:

Asking more qualitative questions is one of the best things you can do as a product manager to gain insights into what your users think of your product. Open ended answers take time to go through, but it’s worth your time to sit down and read through them. In the past, I’ve seen responses from “It’s alright” to others writing War and Peace about their challenges with a product. The amount of insights we were able to collect as a result, was far, FAR higher than simply sending out another “On a scale of 1–10, how likely are you to recommend our product?” question.

To end, remember that if you work on these pain points that users are telling you about in these open ended questions, you will be able to produce solutions that in turn lead to higher recommendation scores, or any other kind of quantitative question that you ask. Don’t be shy. Throw more qualitative questions into the mix.

Like what you just read? You can join the PMPaul mailing list here.

Originally published at www.pmpaul.com on July 20, 2017.

ProductHired Blog

Great Companies and Great Product Managers finding each other

)

Paul Lopushinsky

Written by

I like to write. Mostly about Product Management, but other things as well. https://medium.com/open-product-management http://www.pmpaul.com/

ProductHired Blog

Great Companies and Great Product Managers finding each other

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade