Bad consumer data is worse than no consumer data at all
(& how to avoid bad consumer data)
One of the great things about qualitative user research or design research is that it is accessible, fast, relatively simple and low-cost. You do not need a lot of resources or time to find improvements and to get value out of it. Unfortunately, in user research it’s also easy to bark up the wrong tree, and consequently mistake all your consumer data or findings for valuable insight. And remember, bad consumer data is worse than no consumer data.
Mistaking bad consumer data for insight can be dangerous because it may lead to bad decisions in your product and proposition. There are two main reasons why consumer data can be bad:
- The data is not reliable
- The data is not representative
#1. Non-reliable consumer data
There are a few things that often lead to false consumer data. Or to consumer data that sounds great and you love to believe but is in fact worthless. It has no real meaning or it’s misleading; because it’s not true or because there is no way for you to know if it’s true. And neither will tell you what to do next.
Let me give you an example. Chances are, nowadays there are quite some people with an idea for a product or service related to ‘sustainability’. So, in this case, they might want to know if people are willing to pay (extra) for a product or service like this. How do you go about?
Often people just start asking a lot of people this exact same question via — for instance — a survey. This can give results such as ‘68% of consumers say they are willing to pay a premium for a service that is more sustainable’. This sounds good, but in fact is meaningless and dangerous bad data. Erika Hall, (she is brilliant) is extremely good at explaining the importance and how-to of good design research and has written a great post about the danger of surveys.
It is too easy to run a survey. That is why surveys are so dangerous. They are so easy to create and so easy to distribute, and the results are so easy to tally. And our poor human brains are such that information that is easier for us to process and comprehend feels more true. This is our cognitive bias. This ease makes survey results feel true and valid, no matter how false and misleading. And that ease is hard to argue with. — Erika Hall
If you want to know what people will do, it is better to look at what they have been doing and what drove their decisions, instead of asking them what they think they will do in the future.
Consumers don’t think what they feel, don’t say what they think and don’t do what they say. — David Ogilvy.
When you ask people if sustainability is important to them, the majority will tell you it is. And this might even be true or not. But what does ‘important’ mean to them? How do you know if it is so important to them that they are willing to change their behaviour or purchases?
If you really want to know if what people say is also what they will do you can use the tips for asking good questions from The Mom Test by Rob Fitzpatrick. In short, Fitzpatrick writes that it is best to look at specific behaviour from the past. So instead of asking them ‘Would you be willing to pay extra for this product if it is good for the environment?’ it is better to ask them ‘What have you changed in your behaviour to be more sustainable?’. To make it more specific, it is even better to ask: ‘What have you changed in the last year to be more sustainable?’
Answers to these questions will give you an idea whether someone is finding it important to act and purchase in a way that it is supporting a more sustainable world. You will find out if their actions include options that are more costly or take a lot more effort. That will tell you how important the topic is to them and how much they are involved and committed to live more sustainable. On top of that, you will find the difference between people that are ‘trying to collect waste separately’ to help the environment and people that ‘take a holiday nearby’, ‘become vegetarian’ and ‘drive an electric car’.
Another good way is to ask people about all the things they know they can do to live more sustainable. This will give you an idea about how much they are familiar with the topic of sustainability. Furthermore, I always look for the ‘Moments of Truth’ these are the moments a person started, continued or stopped certain behaviour. An example for a question that will give you an idea about where people started related to the topic of ‘sustainable’ could be ‘What was the first thing you changed to be more sustainable?’ Following this way of asking questions will give you consumer data that is not bad, but in fact reliable.
#2. Not representative consumer data
Now that you are ready and set with good questions, it doesn’t mean you are saved. Even if you do ask the right questions, your consumer data may turn out bad because you have asked the wrong people. How do you know if you are talking to the right and the right amount of people?
One of the hardest things about qualitative research for people is to understand how small samples can be representative? To interview a representative group of people about their experiences with a certain topic, brand, product or service you do not need a big sample as long as you select the right sample. The official rule for the sample size of qualitative research is that as soon as the answers you are getting from your participants about their experiences are getting saturated you have reached the right number of participants.
Rule of thumb: continue the user research until you don’t learn much new from the sessions with users anymore. After that it is becoming a waste of your efforts.
Most UX researchers can make good estimations upfront about the number of participants they will need for their research. From my experience that often means around 8–10 people, as long as there is a clear focus in the research. In some cases, when the topic of the research is very broad or versatile it is more difficult.
Representative consumer data is not only about the right amount of consumer data, it is also about the right participants to get the consumer data from.
How do you know if you got the right participants in your research? The next thing might come in as a shock but contrary to quantitative research the sample does not need to have a similar distribution of demographical and socio-economic variables as there is in society. Whereas in quantitative research it is about the width of the sample, in qualitative research it is about the depth of the sample. In qualitative research the sample should be based on relevant experience. Relevant experience? Relevant experience! It’s the experience that people have that will impact their experience with the topic of the research because there is a direct or clear link between the experience of the person and the experience with the research topic. And this relevant experience is relatively simple to find. Let me give you another example about sustainability.
Again, imagine that you are trying to find out whether people are interested in an app about sustainability. This topic sounds very broad, but is it really? Can it be anywhere in the world? Can it be any type of products? What does this product do? Who is it for? Usually you will know already a bit more about the topic, concept or idea. For instance, you will know that it will be for an app that promotes and rewards sustainable actions and products for customers of a retailer X in Belgium. The goal of this app is to make people more loyal to retailer X.
If you want to select a sample for this research topic you can select them on the following relevant experience:
- Uses a smartphone and has installed and uses apps
- Shops at Retailer X in Belgium (a lot or a little, frequent or less frequent)
- Experience with sustainable actions and or products (a lot, little or none)
If you look for people with mixed relevant experiences, you will likely get a good and representative sample. Usually we add demographical and socio-economic variables for distribution only. So, for instance, to try distributing as much as possible in accordance with the topic to residential area, household, age etc. Fun fact: if you do the sampling based on relevant experience right, as a result it will actually show you which demographic or socio-economic factors are relevant for this topic.
It can be quite hard to find people with the right set of relevant experiences. Therefore, it can be helpful to ask an agency with a large panel to help you out. They can ask the questions about relevant experience to their panel of thousands of people to select the optimal small qualitative sample for you.
Now it’s up to you. Whenever and wherever you go; looking to find valuable insights and improvements for your product and/or services, just make sure to hold on tight to your list of good questions when you jump onto the right people. Good luck!
❤️💡 Interested in more about kick-ass UXR? Check this out! 👇
📬 Follow the IceMobile Medium to receive our latest updates! 🤞👏
