FAQ about ShopBack’s Research Process (Part 1/3)

My team had a great time recently sharing about our working culture in a local design community — Friends of Figma. We enjoyed the session and were heartened by the positive reception. We’re grateful for the opportunity and appreciate the organisers’ help in preparing the session. If you’re keen to watch the session, you may catch it below.

During the talk, we received tons of questions and we couldn’t respond to all of them due to time constraints. We reckon these are good (and hard! 😆) questions. I’ve compiled the questions and shared the answers here. We’ll split them into 3 posts and this will be the first post where we focus on our research process.

Mix Methods

Can you share more about using mixed methods, how do the quant and qual methods work together?

How many % of your research is quant vs qual? Thanks!

Can you share some examples of how your team triangulates qual and quant data? Thanks!!!

If you’re a UX Researcher, you’d have heard these questions before.

“Would xx users be enough as the sample size for the user interview?”

My face every time I hear this question. This doesn’t represent the other researchers in our team.

I have tried multiple templates of answers to address these questions. An example includes highlighting that my approach would be something that is recommended by the Nielsen Norman Group. I have also personally tried to reassure stakeholders that we can always recruit more users if we don’t get enough patterns after one round. At times, it might not be feasible as my team of researchers have 10 other research requests to work on.

Sources of data that we use to triangulate

I realised that qualitative research methodologies such as user interviews or diary studies would never be representative of your entire user base. They’ll never be as statistically significant as quantitative research methodologies. That is why our team has been experimenting with mixing research methodologies for the past year. As we are UX Researchers with stronger qualitative backgrounds, we are still learning when it comes to analytics. All of the quantitative practices are guided by our data analysts, and also monitored closely by our product managers who are actually data shifus and pro researchers in disguise.

ShopBack product managers are actually full-time researchers and data shifus.

It has been extremely insightful so far and it made the entire stakeholder buy-ins easier too. Below are a few approaches we’ve tried.

Diverge: Qualitative Research before Quantitative Research

In this approach, we’ll try to validate findings from qualitative research. Based on the insights that we obtain from the user interview or diary study, we identify behaviours that we can quantify, and we validate them using quantitative data. This can be done through quantitative analyses (in our case using *Metabase and Amplitude), through product experimentations, or surveys. Qualitative researchers tend to be strong in coming up with hypotheses based on the insights we have gathered. A few months ago Amy did a diary study and tried to validate the insights through Metabase, and the insights were really powerful. No one raised any questions on the insights when she presented the findings. However, there have been situations where we invalidated our insights derived from user interviews. As UX Researchers, we don’t just take pride in the insights we produce, but we also recognise that they might only apply to certain users or a segment of users, recruited for the qualitative research study.

*Metabase and Amplitude are tools we use in ShopBack for analysing user behaviour data.

Converge: Qualitative Research after Quantitative Research

There are multiple ways to do this approach. When we use existing quantitative findings that have been done by other teams, experimented by the product managers, or reported by the data team, the findings would inspire the kind of questions we should investigate through qualitative research. Last year, Jasmine did a great job in conducting a segmentation analysis using Metabase. We grouped our power users in a market that we were working on, looked into their behaviours in Metabase, recruited 12 users from these segments, and interviewed them to learn more about their behaviour. The user interview questions were more specific to the topic that we identified from the quantitative findings. However, we also realised that quantitative exploratory analysis can be challenging without establishing clear research questions and objectives.

As suggested by Wei Jian, who is one of ShopBack’s data shifu, not a Samoyed. We are also looking at how we can use SQL to utilise Metabase better to validate some of our insights.

You can read our next post here. Don’t forget to follow ShopBack tech blog for updates!

Huge thank you to Rachel, our content strategist for restructuring and editing my brain fart. If you think my writing is nice, it wasn’t me. I was just farting my thoughts from my brain.

If you have stayed till the end of this post, know that I am super thankful for your time. I hope you’ll take a thing or two away from our experiences! Anyway, I’m always excited to meet new people, so hit me up for coffee or tea anytime. You may also find me in ADPList. 😁

❗️ Interested in what else we work on?
Follow us (ShopBack Tech blog | ShopBack LinkedIn page) now to get further insights on what our tech teams do!

❗❗️ Or… interested in us? (definitely, we hope 😂)
Check out here how you might be able to fit into ShopBack — we’re excited to start our journey together!