Research That Scales: Scaling the Contributions of Others

In a series of articles, we — Cordelia, Jordan, and Erik from the Thumbtack Research Team — share three strategies that we used to scale up the impact of our tiny research team without increasing our headcount. This article, Part 3, is focused on how to empower others to do great research. You should also read Part 1: Scaling ourselves and Part 2: Using past research to answer today’s questions.

—

After streamlining our operations and building a system to make past research easier to rediscover, we had taken our tiny team as far as we could. If we wanted to continue scaling our impact, we knew we had to look beyond our tiny team of three.

We were in an interesting position where we had several teams who desperately wanted more research and were attempting to run their own research projects. From our perspective, that’s fantastic — it means they’ve bought into the value of research. All we had to do was harness that enthusiasm and give them whatever guidance they needed.

In order to help turn these designers, engineers, and marketers into better researcher partners, we designed a two-part training program.

We started by facilitating research workshops for different teams based on the knowledge gaps we observed for each. Since so many on the team had already been doing research without training, we suspected they may have developed bad habits that might be hard to recognize if we only talked about best practices. By pointing out common mistakes and how susceptible we all are to bias, we were able to invite them to make mistakes and accelerate their learning within the safe space of the workshop. We wanted to break them down to build them back up.

We started by simply explaining what research actually is — and isn’t — in order to quickly dispel the myth that research can be easily summed up as simply “talking to people.” It ended up being helpful to be able to level-set and explain not only what we do and how we can help, but how best to work with us. We wanted workshop participants to understand that research is a practice that is hard to master, but is also something that is easy to get started on. And we wanted to be their partners there.

It was important to address the idea of cognitive biases and how avoiding them is such a key part of what we do. We knew we needed to convey how easy it can be to inadvertently influence our findings if we don’t safeguard our study design. We focused on six specific biases to avoid and divided them into two buckets: Biases that impact how the participant responds and those that impact our interpretation of what we heard.

Biases that impact how our participants respond:

• Observer-expectancy (in which a researcher’s own biases subconsciously influences the participant)
• Social desirability (in which a participant answers questions in a way that they believe researchers or other participants will view favorably)
• False recall (in which a participant remembers an event differently than it actually occurred).

Biases that impact our interpretation of data:

• Confirmation bias (in which we interpret new evidence as confirmation of existing beliefs)
• Clustering illusion (in which we see patterns in a random set of events)
• Recency bias (in which we see new information as more valuable than older information).

There are obviously many, many more biases we could have highlighted, but we focused on these because they crop up so frequently in research.

Once we introduced how cognitive biases can influence research, we would have the workshop participants practice moderating techniques. We did this through demonstrations and interactive games, which encouraged participants to try out different ways of asking questions and make mistakes in a safe environment. The idea is to show how challenging it can be to get moderating right while teaching them that a good research question is not necessarily the same thing as a good question that you might ask in an interview.

After the workshops, we offer in-depth one-to-one coaching based on individual needs. We devote around four to six hours of coaching time for someone’s first study and continue to advise them afterwards. We support first-time study owners with debrief facilitation, synthesis and report creation.

We’ve run our two-hour workshop 15+ times at Thumbtack with 200+ attendees from teams across the company. And those attendees quickly put their new skills into practice.

In the last year, Thumbtack as a whole has conducted around a hundred research studies — but only half of them came from our tiny team. Over half of all Thumbtack’s studies were led by someone for whom research is not their primary job. And that’s not even counting all the guerrilla research, product feedback at monthly events, customer advisory council calls, community forum posts, and Tiger Team calls.

We knew throughout the process that our approach wasn’t necessarily novel. We weren’t doing anything that hadn’t been done before. But when we started, it was something that had never been done at our company, and we hope that sharing our story might inspire ideas for what you might do within your own organization.

Do you want to help us continue to scale the impact of the Thumbtack research team? We’re currently hiring a Quantitative researcher and Summer 2020 Interns.

—
This article was co-authored by Cordelia Hyland, Erik Olesund, and Jordan Berry — researchers at Thumbtack. Special thanks to Cory Weaver for the illustrations.