What to do when qual and quant disagree
By Subhasree Chatterjee, Archana Shah, Sanket Shukl and Jason Bressler
In many situations, if different teams are looking at different pieces of data in a vacuum, they come up with different solutions. What do you do in those situations? Who do you listen to? Who “wins”?
As highlighted in our previous article, collaboration is the key! It is not about declaring a winner, or listening to one group, it is about how best to tell a comprehensive meaningful story. If you look at quantitative and qualitative data separately, often the masses tend to gravitate toward and make decisions based on quantitative data. However quantitative data can also be interpreted in different ways depending on what questions you are asking — are you really looking at the complete picture? All the different data sources you might have at your perusal aren’t sufficient on their own in most situations. They all will have nuances, shortcomings and hidden insights. That’s why data needs to be combined to tell a cohesive united story.
We recently ran into a situation where we found customers stated they needed to do something very often, but the behavioral analytics indicated they rarely did it.
What happens next? The risk-averse or quant loving product team sides with the quant data and ignores the problem altogether. The qual-lovers allow themselves to be biased by the users’ earnest and confident responses in the interviews and potentially over-size the problem, sometimes even the wrong one.
Ensuring that your processes and analyses are thorough is necessary, but it doesn’t mean that either side is wrong. We had the opportunity to puzzle over this — simply by constantly asking why. Here is what worked well for us and we strongly recommend: open door number 3.
1. Start by believing both pieces of data are true.
What explanations would make that possible? Et voila! You have new hypotheses. In our case, here were the ones we chose to pursue:
a. The users were doing the things they said they were, but were going outside the product to find answers
b. The importance of doing these tasks impacted the perception of how frequently they did them
2. Go over your data again, together. Experiment more if necessary.
Look at the specifics of the areas of disagreements — there is so much value in asking “I see this. Users might be doing X. Do you see any evidence of that?” Our paths led us down looking through the interviews from different perspectives, digging through behavioral analytics at deeper levels. Even being able to invalidate hypotheses is valuable learning.
3. Present the findings to the stakeholders together.
Layout the entire story together. New perspectives, data points and hypotheses that would have otherwise been left unexplored will emerge and existing ones will be refined. For example, one of ours changed to: they need the information but can’t find it easily, so they look elsewhere.
Side, but important note: we wouldn’t have even known about this if we hadn’t been collaborating with each other. Working together from “go!” is very important and this is just one more reason why.
Sometimes, you only have qual data or quant data, and you have to follow what you see using your best judgment. Other times, the data may be contradictory and irreconcilable, at which point other data sources like marketing, customer support etc. may be able to provide data to corroborate one side or the other. It is very important, however, that you start by looking for a story that explains both sides of what you are seeing .
In conclusion, it is important to keep an open mind and try to look at the problem from different angles and always be ready to pivot, true to the agile and discovery philosophy. Instead of defining right and wrong, embrace the culture of learning.
Thanks to Bryan Campbell and Jeanette Fuccella for their contributions and reviews!
