Reconciling the Differences in Our Data: A Mixed-Methods Research Story

Five steps for turning data discrepancy into better outcomes for customers

Caryn Kieszling
Sep 30, 2019 · 5 min read

By Caryn Kieszling and Sera Mirchandani

A happy customer with a mobile device within a circle composed of letters and numbers (quantitative and qualitative data)
A happy customer with a mobile device within a circle composed of letters and numbers (quantitative and qualitative data)

We all love it when the quant and qual align, but what about those other times, when they seem at odds? For example: the surveys are in, the clickstream data has been analyzed, and you’re feeling confident. Then as you compare notes with your teammates, you realize that the recommended next steps based on UX research and data science are poised to send the business in two very different directions.

This was the challenge we found ourselves working to resolve as a user researcher and data scientist with Customer Insights Research supporting the same product team at Microsoft. What seemed like a conflict ended up leading us to deeper insights, a closer working relationship, and a better outcome for our customers—but getting there was a multi-step process.

Step 1. Confront your data discrepancy

Our product team was sunsetting the older version of an app in favor of one that provides accessibility for all users. To help our stakeholders understand what our customers needed in the new version, researchers had conducted user studies, interviews, and surveys as well as analyzing in-app feedback. Caryn, a researcher, was listening to what our customers were saying: they told us too many of the features they enjoyed in the older app were missing from the new app.

The user research recommendation, based on this analysis? Fill the feature gaps from the older app or customers will not transition over.

Meanwhile, Sera, a data scientist, conducted a cohort analysis with clickstream data to understand what our customers were doing in the older version of the app and how that impacted their transition to the new version. Based on the qualitative feedback, she expected to see customers who used features only available in the older app abandoning the new app. But the analysis showed that they weren’t.

The data science recommendation at this stage? Since customer retention in the new app doesn’t correlate with feature use in the older app, focus on other vital parts of the user journey to help people transition.

Research and data science had arrived at opposing suggestions. Now what?

Step 2. Resist the urge to champion your own data

At this stage, it could have been easy to each double down on our opposing viewpoints. If we’d presented the results, asking our general program manager to choose between recommendations, at least one of us would have the satisfaction of knowing we influenced the product. But how could our stakeholders and leaders be confident they were making the best data-driven decision, if we forced them to choose between quant and qual?

In a way, mixed-methods research is an exercise in getting comfortable with conflict and finding reconciliation instead of a “winner.” Happily, we each realized this and resisted the urge to champion our own perspective. We asked for the time we needed to investigate further, and our product team accommodated.

Image for post
Image for post

Step 3. Dive into your data dissonance

Next, we brainstormed different ways to collect and view our data that might resolve the seeming discrepancy.

Diving into the qual, Caryn investigated which feature gaps were causing people to say they wouldn’t use the new version of the app. Some of the feedback was about features that were built into the new app.

Customers perceived them to be missing because they were in different locations or had different user-interface flows.

Diving into the quant, Sera took a closer look at who was transitioning completely. She created a new category of retention in the cohort analysis — new-app-only retention. If a customer used both the new app and the old app, they were retained in the new app, but they were not retained in only the new app. She found that although customers did use the new app, many would only do so while simultaneously using the old one. Furthermore, usage of specific features in the older app did correlate to whether customers used the new app alone.

Step 4. Synthesize data to align on a new recommendation

As we synthesized these new insights, we found some overlap between Caryn’s list of perceived and actual feature gaps and Sera’s list of features that correlated with an incomplete transition. The overlap revealed which features were making it most difficult for customers to transition. Some of these were really missing in the new app, while others were perceived to be missing.

Putting the pieces together, it looked like our customers would need the following to make a complete transition:

1. In-app onboarding help to find features that had been moved

2. A larger set of features

Only by providing both resolutions could we offer our customers a seamless, complete transition to the new app. Interestingly, our final recommendations weren’t so very different than the original “conflicting” ones — only now we understood the subtleties of how they fit together.

Infographic showing the methods used to reconcile data sets
Infographic showing the methods used to reconcile data sets

When we shared our new recommendations with the product team, it turned out that some of the actual feature gaps we’d prioritized were impossible to bridge in the new platform. Thanks in part to our data, our stakeholders decided to continue supporting the older version of the app.

Step 5. Seek further opportunities to collaborate

We know just how disconcerting it can be when data sets compete. But as data and customer stewards, it’s our job to use these moments as opportunities to play off each other and get at deeper insights. Facing our discomfort head on helped us build a better partnership.

Mixed-methods research has become a necessity for our UX work. It’s the friendship between qualitative and quantitative research that ultimately gives us the best chance to do what’s right for our customers.

How do you collaborate to achieve richer UX insights? Let us know in the comments, or tweet us @MicrosoftRI! Want to work on mixed-methods research with us? We’re hiring!

Microsoft Design

Stories from the thinkers and tinkerers at Microsoft

Caryn Kieszling

Written by

User Experience Researcher at Microsoft. I study how humans perceive, understand, and interact with technology. My views are my own.

Microsoft Design

Stories from the thinkers and tinkerers at Microsoft. Sharing sketches, designs, and everything in between.

Caryn Kieszling

Written by

User Experience Researcher at Microsoft. I study how humans perceive, understand, and interact with technology. My views are my own.

Microsoft Design

Stories from the thinkers and tinkerers at Microsoft. Sharing sketches, designs, and everything in between.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store