Qualitative vs Quantitative research: Which is better for UX?
There are two main types of UX research, qualitative and quantitative. What are these, and which is most useful when designing usable products or services?
You are probably expecting me to say ‘it depends…’ but I do actually believe one is more useful than the other. And I’ll explain why:
What is QUANTitative research?
Examples: Usage statistics, A/B testing, NPS score, star ratings.
Quantitative (sometimes spelled Quantitive), from the Latin ‘quantitas’ or in English ‘quantity’, refers to learnings derived from data, such as trends from tools like Google Analytics or Power BI.
Quantitative data tells us what people are doing.
What is QUALitative research?
Examples: Moderated user testing, interviews, diary studies, customer feedback.
Qualitative, from the Latin ‘qualitas’ or in English ‘quality’, refers to learnings such as customer quotes, or observations of people using your product or prototype. Most commonly this might be a user test where we see what they do and ask them questions.
Qualitative findings help us understand why people might be doing something.
Some methods are both
Some methods can be either qualitative or quantitative depending on how you use them or what you ask. For example, if you ran a survey with the question “how often do you purchase cheese?” the output would be Quantitive — we would be able to get a figure on the average frequency of purchase.
However, if we asked “why do you buy cheese” and provided an open text response, this will give us qualitative answers.
Of course, we could group those responses, and see if there are any trends or common reasons given, and then we would have qualitative evidence for cheese purchasing triggers.
Do you need both?
It is ideal to have both types of evidence as there will be different questions you can answer at different times with different methods.
For example, a large company might employ data analysts. Let's say one of them notices there is a high level of dropout on a certain webpage. This data is Quantitive, and it shows us that there might be a problem. It might even be able to be segmented to show that this is for a certain customer segment, or during a specific action, but it doesn’t tell us why.
To answer why people are dropping out we need a qualitative method. For instance, we might use session recordings to see if we can spot what is happening. Once we have an answer we can design a solution.
Before we spend time and money building our proposed solution we would want to test a prototype to see if it solves the problem. To do this we would likely run a qualitative usability test.
Now we have validated our solution, we can build it and release it. Most large companies do ‘staged releases’. This is when they release the change to a small number of users to make sure it doesn’t cause any issues. This is really a Quality Assurance process, but we can use this as an A/B test.*
A/B testing is a quantitative research method and can tell us if our solution works in the real world.
*Sometimes referred to as split testing or multivariate testing. Though the latter is subtlely different.
So which is better?
I hope I’ve made it clear that both are ideal.
However, not all companies *can* do both: Quantitive research needs a great deal of data to get statistical significance.
For AB testing, for example, that often requires hundreds of thousands of sessions on a webpage. The reason is…
Correlation does not imply causation.
If I flip a coin 10 times and it comes up heads more often than tails it does not mean I’m more likely to see heads in the future. I need a lot more flips to get statistical significance.
I recently consulted for a client who have thousands of customers but we were still unable to use much in the way of quantitative data to answer the questions we had because there simply weren’t enough to get statistical significance.
This takes that option off the table for all but the biggest of companies.
However, I can use qualitative research to get an idea of how big a problem might be. If I run a test with 5 participants and the same thing happens with 3 or 4 times, it is fair to assume this might be a big problem in the real word too. We can’t know for sure, but we have evidence that it might, and more importantly we likely would have a better idea of why it is happening, which we wouldn’t with a purely quant method.
Thanks for reading. If you would like to be able to combine your research to be able to deliver better UX, please give Glean.ly a go.
