The Harmony of Qualitative and Quantitative Research
Hi! I’m a Design, Technology, and Innovation Fellow for the City of Austin. I work with a group of awesome humans on the Fellows’ Austin Resource Recovery project researching behaviors and perceptions of residents using services. We are using human-centered research, design, and rapid prototyping to develop solutions that positively impact this service. You can find information about the fellowship here.
I am an analyst at my core. I generally prefer objectivity to subjectivity, convergence to divergence and safe certainty to bold uncertainty. I’m very open to new ideas, especially those rooted in data, but if you told me a month ago that quantitative data could not tell you the whole truth, we might have to agree to disagree.
I was hired onto the City of Austin Design, Technology, and Innovation fellows team last month to help support the qualitative design research project for Austin Resource Recovery. My most recent work has been in User Interface design and prototyping, but I have a background supporting user research from a quantitative angle. I’ve led research projects to support design, but they were focused on measuring website performance using user segmentation to optimize discrete flows. These are fairly precise and usually very linear processes, lots of “if this then that”, and the outcomes are narrow.
I popped into this qualitative research project during the synthesis phase, having very little exposure to the qualitative side at all. So this is my first time on this journey in a very intense, immersive and real way. I’ve learned a lot this month and it’s reshaped the way that I think about data. My goal in this post, is to share with my quantitative peeps, why qualitative research is the jelly to our peanut butter.
Smaller Bites
With qualitative research, you need a much smaller sample size to extract something true out of your dataset because of the sheer richness of each data point.
I was surprised to learn at the beginning of our ARR research, that our sample size was 48 people to represent a population of over 800,000. But think about it! When you narrow a big dataset down to what you can actually use for an analysis, you may have millions of rows, but you narrow it down to a few columns that you want to analyze.
Qualitative datasets are kind of like big data turned on its side. Debriefing is a design research method where you break down deep field research (in our case 90 minute in-home interviews) into smaller data bits. Each data bit is a quote, story, or observation made during the interview. Synthesis is the process of reconstructing qualitative data bits into insights. During synthesis, you are essentially tagging this data, or creating and assigning meaning to columns that it can fit into. So if you think about it this way, qualitative datasets have a huge number of columns, so with just 48 rows you still have a very rich dataset.
That’s just if you’re being totally numerical about it. There’s also the fact that because the researcher has to be present for the collection and synthesis of every single data point, they end up knowing the data much more intimately than if it were just running through an algorithm. You capture data that’s hard to record like body language, tone, whether or not there is background noise, or children making noise in the background forcing them to rush their answers.
Cut the Crusts Off
Since qualitative data is unstructured as it comes in, there is no set workflow or process for analyzing it. You build your process as you go, depending on what you find, and what’s working. It can be uncomfortable to not know what’s next, but if you can get past this (and trust me, if I can, you can) you can reap quite a few advantages from it.
One advantage to qualitative research is that since you’re taking in and processing information simultaneously, you usually have insights that develop very early on, often way before you process the whole dataset. This lends itself very well to co-creation because it means you can gut check with your stakeholders and end users along the way, and involve them in iterations. In the ARR project, we developed a persona after just the first week of synthesis. This persona was “the analyst”, and she was derived from the insight that without proof of, or connection to the greater impact on recycling economics, this person would not go out of their way to recycle or compost. She would, however, recycle if systems were present. We thought about this analyst as having systems but lacking motivation. This persona, along with a couple other personas and insights, helped us to articulate three abstracted components of recycling behavior that we believed were essential to successful recycling: motivation, systems, and knowledge.
Some discoveries jump off the pages and some of them hide for weeks, but the value of early findings and the opportunities they give is very unique to qualitative research.
Soul Food
There was more than one example of us learning more about our recyclers than they previously knew about themselves. This is a depth of insight that quantitative data can’t give you. The first example of this came from an interview with Eric. He shared his experience living with roommates who don’t recycle, “talking to you makes me realize that I feel alone.” This supported a quantitative discovery from within our sample: the more people in your household, the more likely you are to recycle “never.” On its own, it’s a bit dry, but coupled with the understanding of the “social support” motivation, it becomes a very cool story. There is also the element of self-reported behavior versus actual behavior. Usually, data analytics either measures one, or the other, but this can lead to incomplete pictures of a person’s behavior. An example of this in the ARR study is Roger, who reported that he recycled “never”. This was true when he was at home, but at work, he did recycle. . In fact, as the office manager, he was the one who ordered recycle bins for the office after months of nagging from a co-worker. If we didn’t have the in-depth conversation with him and probed deeper into his habits, we would have just taken his self reported data at face value. Instead, we uncovered the power of social pressure, which became one of the key insights. at work. Qualitative research cuts deep, and therefore, you gain a more layered understanding of the data, and therefore, the subject matter.
It’s Peanut Butter Jelly Time!
I didn’t talk about the advantages of quantitative data because I wrote this for people who have all their eggs in that basket already. I don’t expect anyone to jump to the other side of the fence, I just want to recommend that if you are investing in these projects, that you consider both.
When you have qualitative depth of 48 people, you can decide what behavior you care to verify, and how to collect a representative sample for a population of 800,000. If you can develop strong discoveries through synthesis you can form strong hypotheses to test in this sample. And, as an analyst, when you understand the many layers, contradictions, misunderstandings of quantitative data to represent human behavior, you can empathize with your data points, see your dataset for what it is, and ask it the right questions.
I’m not saying don’t eat the peanut butter sandwich. I’m just asking that you try jelly.
