Mixed Methods Research

Mixed methods research combines multiple data sources and data collection methods, typically both quantitative and qualitative, to examine a phenomenon from a diverse set of perspectives, so that we learn not only what took place, but why and how.

Let’s look at some reasons for augmenting quantitative data, such as those collected in the first Context Graph experiment, with qualitative data. We’ll also introduce a few studies conducted over the past few months on user browsing behaviors and content discovery on the web.

Quantitative vs. Qualitative Data

The first Context Graph experiment is a collection of quantitative browsing data.

For example, by examining the time and duration of webpage visits, we can compile the statistics on concurrent tab usage and understand what took place in a user’s browser. However, such numbers alone do not answer questions such as why, how, or in what context an event takes place. Why do some users browse the web using only two tabs, and others twenty? What goals do our users have in mind when they start Firefox? How often do they successfully complete their tasks?

As we move towards our goals of understanding how people browse the web today and improving your experience with Firefox, we will deploy a variety of research methods, depending on the nature of our hypotheses and analysis goals. In particular, qualitative methods can be applied to study user intents or behaviors difficult to discern from numerical data alone. Qualitative methods can also gather certain data more efficiently with less privacy intrusion. Finally, while the detailed statistics from quantitative data can help us optimize the solution to a problem, qualitative data can be more effective in identifying key problems that our users face. Below, we elaborate more on our motivations and highlight some potential benefits of a mixed methods approach.

Motivations

Consider the following research question: Do users who browse the web using more tabs also have better computers?

Answering such a question may help us identify hardware bottlenecks and optimize Firefox for certain system configurations. If we had recorded the hardware profiles of our study participants (we did not in the first Context Graph experiment), we could test this hypothesis by examining the correlation between computer specs (e.g., CPU, GPU, memory, network, monitor) and tab usage.

All the above variables have a natural ordering. Four CPUs provide more computational power than two; 16GB of memory more capacity than 8GB; dual monitors more screen real estate than one; and so on. In other words, we have clear indicators of what constitutes a better computer and what constitutes greater tab usage, and can directly calculate the relationship between the variables.

Other analyses, however, can be more difficult to answer with numerical data alone. For example, what the types of content are available on the web today? How do they compare to available content from two, five, or ten years ago?

We could apply machine learning techniques, and build a statistical model that categorizes all webpages by content types. However, such a classifier does not answer broader questions such as: Do these machine-generated content types make sense to a person? How many categories of content should there be? If two algorithms produce two competing sets of categories, how do we determine which one is better, more suitable, or more meaningful?

Benefits

Transparency, Interpretability, and Trust

One issue with approaching a problem with strictly quantitative methods is the lack of transparency in the analysis, as exemplified by What We Can Learn From the Epic Failure of Google Flu Trends.

When Google Flu Trends were introduced in 2008, Google researchers found that they could predict flu prevalence based on the volume of search queries. While the statistical model was initially accurate, its inner workings were opaque to the model builders, engineers, and users. Aside from producing a likelihood estimate, the black box predictor could not explain why a search query such as “high school basketball” affected its calculation. When the model failed in 2013, the lack of transparency makes it difficult to explain why the data-driven approach failed, when it would fail again, or how to maintain its accuracy over time. The project was quietly killed.

As for the Context Graph project, building a statistical model with usage data may help us predict when a user will visit a news site. However, such a model alone does not answer questions such as: why a user is looking for news, whether the user has difficulties finding relevant news, and how to improve the user’s experience.

Guiding Principles, Explanations, and Privacy

A second benefit is that we could actually answer some questions more effectively and in less intrusive manners, by applying mixed research methods.

For example, rather than collecting all browser usage data in order to infer user behavior, we could directly ask the users: what they want to do and whether they accomplished their tasks. Users can describe their intents (e.g., in a survey), in ways that opportunistically-collected data cannot. Users also have the option to decline answering any questions, and are free to share as much or as little information as they wish. Such approaches do not scale as well as quantitative analysis, because they are costly to analyze. However, they can yield useful insights and can be very effective early in the analysis process.

Finding a Solution vs. Finding a Problem

Finally, quantitative and qualitative research methods differ in that the former is better at finding an optimal solution to a problem and the latter is better at finding a problem.

Former Google designer, Douglas Bowman, once fretted about ineffective use of A/B Testing on the wrong tasks when resources were spent optimizing the solution to a problem (Which shade of blue will maximize user clicks on a toolbar?) even though it is unclear whether a problem existed in the first place (Does the color blue matter? Do users even want a toolbar?).

This dichotomy is sometimes phrased as getting the right design vs. getting the design right in the field of Human-Computer Interaction. We would very much like to ensure we understand the difficulties people around the world have in freely accessing the web, so that we address the right problems rather than merely addressing a problem the right way.

Next Steps and Your Impact

We’ve been collaborating with the Firefox User Research Team who have extensive experience in qualitative user research methods. We are also leveraging various tools developed by the Strategy and Insights Team including Heartbeat and Shield, to deploy surveys, in-context studies, as well as lightweight data collection.

With these tools in place, we conducted various studies over the past few months:

• Multi-Tab Browsing Behavior: How do people organize multiple tabs, multiple windows, and even multiple browsers as they navigate the web?
• Page Interactions: What tools, interactions, or content do people find helpful and unhelpful on a webpage?
• Discovery and Navigation: How do people first learn about a webpage, navigate to the page, and return to the page over time?
• Workflow and Follow-up Actions: How does a page visit fit into a user’s workflow? What actions do people take if a page is helpful to their task?

Finally, we are also collaborating with various product teams to disseminate what we learned, determine what to investigate next, and explore ways to improve your experience with Firefox.

Your participation in the Context Graph studies is contributing to our work across the Mozilla organization. We will keep you posted on our progress. (Expect a few more blog posts on the above studies to come out soon.) We also plan to launch various studies in the coming days, including a second Context Graph experiment and other smaller studies. You may be randomly selected to participate.

We hope you will continue to engage with us and help us create a better browser and a better web experience. Please let us know at any time, if you have any feedback, comments, or questions!