Using Qualitative and Quantitative Data to Answer Ultimate Questions

There is sometimes a moment of shock when quantitative researchers are presented with a strange and unexpected number.
In the Hitchhiker’s Guide to the Galaxy, a group of leaders have built a giant supercomputer to calculate the answer to ‘The Ultimate Question’ of ‘Life, the Universe and Everything’. It’s such a difficult problem that the computer will have to work on it for 7.5 million years. Undoubtedly the researchers are taking the trendy big data approach: process all the data you have, whether relevant or not. They then deal with the scale of the dataset with lots of processing power and advanced machine learning.
Yet when the computer is finished, the answer it shares is just the number ‘42’. This is met with great disappointment to the great-decendents of the original researchers, and to confusion from the rest of the galaxy.
Douglas Adams’ joke about our yearning to understanding our existence is still as fresh a trope today as it was at the time (40 years ago!). But it hides two very important nodes of caution for quantitative and statistical researchers. The first, make sure you ask the right question. And the second? Make sure you have enough context to understand the answer.
In the Hitchhiker’s Guide to the Galaxy, when the researchers complain to the computer about the answer it gives ‘The Ultimate Question’, it retorts that the question was too vague, and the researchers were not really sure what they were asking. It then states that if they want to know what the Ultimate Question actually is, it will take 10 million years, and a computer the size of Earth.
And that is the nub of the matter — knowing the right question to ask is often more difficult than actually answering the question. But this is also the strength of starting your research endeavour with qualitative methods. You can use a very small sample size, and do some ‘quick qual’ asking open ended questions to a few people to pull apart the difficult issues and how participants frame them.
It’s no use asking questions as part of a big quantitative survey if respondents don’t quite understand what you are asking, or if that isn’t the significant issue for them. But how do you start designing something with enough reach and significance that will convince a client on a trend or strategy?
Researchers who are spending a lot of time on a project working closely with clients can become experts on a product or service and the terminology. However, not all users have the same level of knowledge, and may use different non technical language. Asking about a ‘beverage holder’ in a car might be baffling to passengers who just call it a ‘cup holder thingy’. So a small qual study can quickly show you not just the issues that are important for customers and participants, but their ‘native’ language that you need to engage with.
Similarly, clients and researchers may have preconceived ideas about the issues that participants have and want to investigate. Often that’s the brief, and the end of it. But often the most valuable insight is illuminating areas that weren’t even on the radar, and starting with a more open-ended approach can unlock this.
But that’s also the balance to get right. Big quantitative insight works best with a few, well targeted questions. People hate filling in questionnaires where they have to rate dozens of items on scales of 1–10, so to get the best engagement rates, and thus the best reflection of the whole population, you have to focus on the important questions, and not a scatter-shot approach. Fortunately, small qual precursor studies will help you refine this, and get the right questions with the right language.
Qual is expensive to recruit, and to analyse. But it also doesn’t require large samples, so total costs can be kept reasonable. And it can be very dynamic and responsive. Use some qualitative analysis software like Quirkos, and you can quickly learn from and iterate question generation that feeds into the main phase of the research.
And once the big results are in, you might go back to small qual to contextualise unexpected quantitative findings. Just as in the example above, being presented with the number ‘42’ without context makes the result meaningless. All is well and good when a statistic confirms a hypothesis or trend, but for the result that shocks and baffles, it’s time to drill down into that number with more qualitative data.
For example, when a particular factor is ranked unexpectedly badly by participants (or well, but this doesn’t seem to worry clients as much), small qual can come to the rescue. Now you know the exact quantified pain-points for participants, and you can go back to a small sample asking open ended questions to get the ‘why’ and the ‘what next’. This doesn’t have to be a big outlay: a few verbatims from a couple of people can really turn a presentation from ‘Huh?’ to ‘Aha!’ moments.
That’s why my ideal qualitative research projects always go qual > quant > qual. Getting the questions right, getting scope and scale, and then the final context for presenting the findings. That way, the next time clients come with seemingly impossible questions that encompass Life, the Universe and Everything, we can give answers that have as much meaning as the question.