Mixed methods in design research

Combining qualitative and quantitative approaches

The goal of most design research is to understand a problem better, so that we can design a better solution. The specifics of the problem, and what we already know or don’t know about it, are different for each project. But if the problem is worth solving, there are always people affected by it who don’t yet have a great solution. Understanding who these people are, and why they need a better solution, is the first goal of most research projects. Knowing how to design a better solution is the next goal. A thoughtful approach to research allows these goals to work together, and to let the answers to one question help us decide which question to ask next. For digging deep into complex problem spaces, a mix of qualitative and quantitative research approaches can provide a robust and realistic perspective.

First, some general definitions:

Quantitative research: quantitative research techniques are used to observe or test something in a measurable way, with an output that is generally in some kind of numerical form. There are many very different quantitative techniques, ranging from counting numbers of things to complex statistical comparisons.

Qualitative research: qualitative research methods are generally focused on specific experiences, often through some form of personal interview or narrative, and typically generate non-numerical results. Common qualitative techniques in design research include contextual analysis, ethnography, structured interview, and observation.

Mixed methods research: mixed methods approaches combine quantitative and qualitative techniques to gain a broad perspective on a problem. The most effective mixed methods approaches deliberately take advantage of the strengths of each approach in context of a particular problem.

Much of my research experience comes from my work as a cognitive neuroscientist studying biological origins of psychiatric disorders. While much of the statistical methodology was quantitative, I kept a strong focus on understanding the human experience and the personal impact of biological problems, and capturing information at that personal level required qualitative research methods. As with research that has a clinical focus, design research at its heart is about human problems. It is often easier to display and discuss quantitative results, but as a discipline focused on understanding human experience, and it is important to give qualitative information a voice in interpretation of research results.

Mixed methods research approaches allow us to get the most impact from combining qualitative and quantitative research methods. I use a few different mixed methods approaches depending on the stage of a project:

1. Defining problem narrative & forming a research plan

2. Synthesizing trends across a project

3. Detailed data analysis

1. Forming a research plan & narrative

Exploratory and explanatory sequential research designs are both terrific for exploring a problem that has a human-centered core, and cannot be captured with numbers alone.

Exploratory Sequential Design

Exploratory sequential design starts with qualitative research such as ethnographies or user narratives as a basis for defining questions & hypotheses. Quantitative data collection and analysis are then used to test hypotheses and determine a mechanism for the issues uncovered by the initial qualitative methods.

When I ran a brain imaging research study I combined brain image data with behavioral and clinical measures. I also included a structured interview that focused on psychiatric conditions, and nearly always turned into a vibrant discussion about feelings and experiences. Beyond facilitating a rapport with my participants, I was able to learn a lot about the things that bother them, and in turn understand what questions I needed to address to develop better solutions to their problems.

In a product setting, we often start our research with qualitative approaches to understanding a problem. When I was doing initial research for the Slingshot App to understand how competitive foosball players could use an app to facilitate foosball practice, I started by interviewing foosball players to get a sense of what frustrations they had with practice sessions, and what they currently used to address their frustrations. The insights gained from these interviews gave me an idea of what app features would be the most useful to the players, and a direction for quantitative research approaches to test how much impact each feature might actually have on player practice.

Explanatory Sequential Design

A related approach is Explanatory Sequential Design, which is the inverse to Exploratory Sequential Design, in that it begins with analysis of a quantitative data set. Trends and associations from the quantitative analyses are then used to narrow down hypotheses and develop qualitative approaches to uncover explanations and possible solutions to the problems described by the numerical information.

An example of this approach is found in a research plan that I contributed to Wikimedia Commons to address low retention rates of new Wikipedia contributors to the Indonesian Wikipedia. I was provided access to quantitative information describing contribution statistics for every Wikimedia project, including contributions to each language Wikipedia. I used a variety of statistical analyses to understand Indonesian Wikipedia contribution details, and uncover basic discrepancies in expected contribution rates based on rate of mobile versus desktop site usage and expected ratio of mobile versus desktop editor contributions. I paired these findings with a qualitative assessment of Indonesian culture, including the nearly predominantly mobile internet access in the country, to hypothesize that contributions to the Indonesian Wikipedia came from 2 distinct group of editors — those contributing from within Indonesia and those contributing from outside of Indonesia — with different new editor retention rates from each group. My plan then described a course of qualitative research approaches to test this hypothesis and explore the potential reasons behind the discrepancies.

Hypothesis Funnel

The “Hypothesis Funnel” isn’t a separate method, rather a combination of exploratory and explanatory sequential design that describes the general outline of most of my design research. This research approach begins with qualitative assessment of problems faced by users, followed by quantitative tests of hypotheses and possible solutions. The project is concluded with qualitative assessment of the user experience after solutions are implemented to assess if our goals were met and ensure the success of the project.

The “Hypothesis Funnel” combines qualitative & quantitative research approaches

2. Synthesizing trends across a project

I use a few different techniques to interpret data and synthesize trends from qualitative and quantitative data across a project. These techniques are perfect when you have a mix of quantitative and qualitative information, and can be especially helpful when results from different research approaches are seemingly contradictory.

Triangulating results

Triangulating results from different approaches to the same problem brings together interpretation of different types of data from a project. In general this process begins with summarizing and listing results from each method used, and noting where results from each method support similar conclusions, and pointing out where the results are not in agreement. The largest value of this method typically arises from exploring the factors leading to disagreement between different research methods, and can uncover important factors not previously considered.

Case matrix

The case matrix method is useful when there is both qualitative and quantitative information available for at least some portion of research participants. The technique focuses on data cases rather than aggregations, and begins by creating a matrix for each case which can summarize the results from each research approach. A second matrix can be constructed for all cases, with rows of participants and columns of research approaches. These matrices allow a detailed assessment of similarities and differences between approaches and patterns across cases.

3. Detailed analysis techniques

There are a variety of methods that allow statistical combination of qualitative and quantitative data. Many of the methods that I use involve coding qualitative information so that it can be essentially treated as quantitative information. There are also many statistical tests designed to be used with categorical data, which some qualitative data fit nicely into.

One of the things I love about design research is that there are so many tools available to uncover and explore human problems and possible solutions. Combining qualitative and quantitative approaches provides real power through probing a problem’s boundaries, origins, and explanations, while maintaining focus on the people who are affected by the problems we study.