Insight at Scale
I talked with Tatiana Gabor from Spotify about the collaboration between user researchers and data analysts and how to combine qualitative insight with quantitative data.
Dessy: Can you tell me about your experience with quantitative and qualitative data?
Tatiana: Most places where I have worked, have user researchers that do qualitative research separately from quantitative analytics group, even though the two teams do overlap. For example, in product development, the process usually starts with user research conducting consumer interviews or/and surveys. Also depending on the product or feature, there could be a separate team that analyzes high-level marketing trends. Once in a while product analysts like me will step in during this first stage to get some insights from existing data to help understand consumer needs.
Usually, though, I do not get involved until the product, or its first version is built. After the initial building phase is complete, the product or feature is rolled out to a small number of users and I run A/B tests. I usually look if any of the important metrics in the test group that has gotten exposed to the feature/product has moved. Based on the results, the feature is either released as is or it undergoes another iteration. Once the product is released to everyone, I will usually track its performance on some dashboard. I guess I should let you actually ask me questions before I unleash all this info on you.
Dessy: You’re doing great! Can you talk some more about those overlaps you have with the user researchers?
Tatiana: Hmm, well for one, I have worked with user research to hone in on the users that match specific characteristics for interviews. Sometimes there is a need for users in a specific age bracket, in a certain country etc… I guess more interestingly, I could see a trend in the data that is not intuitive or is hard to explain. Say a change in UX is implemented and I see a drop in a KPI (key performance indicators). That’s bad. I want to know why. Here is where user research will come in to interview people to understand what actually happened.
You can miss a lot with data, users can be upset but it won’t be reflected in the metrics. And with user research, the results can be easily biased. So the two not only complete but also enhance each other.
Dessy: So you’re using quantitative methods to discover something interesting and then use qualitative methods to explain why it is happening, right?
Tatiana: Exactly. That is definitely a popular use case. I personally believe that user research and analysts working together can be more powerful than the sum of two.
Dessy: Does it ever happen the other way around?
Tatiana: It happens the other way as well. User research uncovers an interesting insight by interviewing a small sample of users and I can look for quantitative proof of that insight in data. I think that data analysis and user research on their own are imperfect. Like you can miss a lot with data, users can be upset but it won’t be reflected in the metrics. And with user research, the results can be easily biased. It can be hard to get a representative sample, especially with in-person interviews. So the two not only complete but also enhance each other.
Dessy: Can you think of an example that you can share?
Tatiana: An experimental change in UX was made that did not seem to move any of the performance metrics. There was no indication that it made customers more engaged or committed. So after some time we removed the feature as it was not worth the investment. We didn’t see any difference in metrics, but there was a huge backlash from users in user research interviews. They loved the feature we just did not see it in our data. But then we decided to look deeper into data at a metric we were previously not looking at. And saw that this feature was indeed very popular and highly engaged with. So user research helped us measure engagement properly.
Recommended Reading:
Hybrid Insights: Where the Quantitative Meets the Qualitative
Human Centred Data. Using Data in a Human Centred Design Process.
