How Thumbtack Built Their Research Ops Function From Scratch

The saying “Cooking is an art, baking is a science” is believed to have originated in a 1967 King Arthur Flour catalogue. And although this simple quote is obviously not always true, it is a helpful shortcut for understanding the key difference between the two culinary practices: Cooking generally allows for less precision and more improvisation. Baking, however, is heavily dependent on specific ingredients, gathered in the right proportions, baked at just the right temperature and time.

“User research is like cooking a meal. Research Operations is like baking a cake” is probably an even less accurate saying, especially given how much science goes into producing robust and reliable research! However, at Thumbtack, we’ve found this metaphor to be helpful in describing the different approaches that these functions need to take to drive impact.

Researchers generally have the ability to adapt their recipes for success in the moment, the way a chef can swap out certain ingredients and still create a delicious meal. If a researcher’s prototype fails to load during a concept testing session, they can pivot and have the participant test the live app instead.

For a baker, however, there is no substitute for something like baking powder (unless you want to eat a cake that is flat and dense.) The baker must ensure that the essential components for success are always at hand. Research coordinators, recruiters, program managers — whatever we call ourselves — are like bakers. For us to be impactful, we need to first ensure that our team has the essential components for success before we add the frosting on top.

Thumbtack has had a user research practice for many years, but it wasn’t until I joined in 2018 that we established our research operations team. As a team, the first thing we did was define the following: what are the essential ingredients that you need to make research happen? At this point we didn’t ask (yet) what could be done to make us more efficient or effective. We just wanted to establish the basics. The foundation for our team.

In this article I’ll cover how Thumbtack’s research team baked a research ops cake layer by layer, first focusing on the foundation needed to make research happen, then adding additional layers to optimize our tools and processes and best support our cross-functional partners, and finally, deciding what the final icing on the cake should look like.

Layer One

Make sure your team is legally compliant. Put basic tools and processes in place to collect data, and make sure you can reach participants.

Our number one priority was making sure we were following the law with our research practices.

We partnered with the legal team to draft an NDA for participants to complete prior to interviews taking place. Then, we needed a way to find participants to speak with! Our first solution was pretty ad-hoc and messy. It involved pulling users from our database, sending out research email invitations, manually scheduling interviews, and then tracking all of this information in multiple Google sheets owned by different people.

For data collection, we already had access to Zoom and Qualtrics at a company level, so we were able to conduct and record remote sessions, and build out basic surveys. All of these things were absolutely necessary for research to happen.

At this point, the first layer of cake was baked. We had enough processes in place to conduct research legally, but we had a long way to go before this cake was really delicious.

Layer Two

Make research effective and efficient. Find the right tools to target specific users, gather better and different types of data, and analyze that data in the best way.

Our second layer and the next area of focus for our team involved finding a way to connect with the right kind of participants, and also gathering & analyzing the data from our conversations with them quickly. Building this layer involved incorporating new and more advanced tools to our suite. We broke them down into each of these objectives:

Our team needs tools to…

• Find the right participants
• Gather more types of data
• Analyze the data

Finding the Right Participants

The first tool we used for this was UserInterviews. This worked great for reaching participants outside of our database, allowing us to hear fresh perspectives of those not already on the Thumbtack platform.

From here, we brought on Ethnio. Acquiring this tool made a huge difference in our participant recruiting process by improving our response rate and reducing our no show rate, making it exceedingly more efficient.

Ethnio allowed us to do things like add Thumbtack branding on all communications, increasing the likelihood of someone opening an invite email. We could now also send automated, customizable reminders to participants which included the Zoom link to join the session, reducing missed sessions, and also offer multiple types of incentives as an added bonus for our participants.

Using a combination of UserInterviews for customers not on the Thumbtack platform, along with Ethnio for reaching our own users, greatly streamlined the recruiting process, allowing us to focus more time elsewhere.

Gathering More and Better Data

Next we thought about gathering more and better data. Here, we recognized the research team was blocked from doing their most impactful work by having only the ability to conduct moderated studies, and therefore a limited variety of data types.

Because of this, we decided to add in UserZoom Go (previously Validately).

This was great because it allowed us to conduct different types of research with unmoderated tests that are simple to set up and yield fast results. This tool is also user friendly enough for non-researchers to run studies, generating more insights across the product teams.

The second improvement we made in this area involved acquiring the MaxDiff feature within Qualtrics. This allowed us to better measure our users’ preferences and answer questions such as: What is most important? What do users most prefer? Investing in this tool allowed us to build more robust surveys and saved the company time and money by sometimes replacing experiments and A/B tests.

Analyzing the Data

Now that we were reaching the right participants and had a way to gather data effectively, our next step involved putting the right tools in place to analyze that data and surface insights.

Reduct is a great video transcription tool that we added to our suite. Researchers are able to receive human-generated transcripts of their interviews a mere hour after uploading an interview recording, tag the data within the transcript, and create highlight reels or snippets of user insights for share-outs with other teams.

We also recently added Mural, which is a remote collaboration tool. This allows for observers to take notes in a centralized place during the session, and teams can use these virtual “sticky notes” in synthesis activities as well.

While the tools we just described are integral resources for any research team, I also want to call out the importance of having strong working relationships in place with other “bakers” in the company, like your legal and IT teams. Having these strong relationships helped maintain our optimal environment for baking and made procuring new tools a smooth process with clear benefits across the company. Having thorough communication and a strong business case in place for each requested tool when looping in these other bakers makes all the difference!

Layer Three

Enable cross-functional partners to find and engage with research.

The third layer of our cake focused on our cross-functional partners. We wanted to enable them to engage with research, both as it was happening and after it was done. We wanted to help them find what they’re looking for, participate in research sessions, and feel empowered to run their own studies.

As both the ops and research teams scaled, and more research than ever was being conducted, we were faced with a new challenge: we didn’t have an organized way to store our findings. This led to inefficiencies like duplicated research and slow information finding.

To solve this problem, we ended up building an internal “Research Archive” to catalog and organize all our research reports and literature reviews in one centralized place. Now the final step for every research study is to add it to this archive.

We chose to build an archive for research reports rather than a repository for research insights or raw data. After some early prototyping and testing internally we learned that being able to create new knowledge by recombining existing insights wasn’t as important as simply having one place to go to rediscover past projects.

We used a tool called Coda to create an easy to use yet powerful database of projects that is accessible to anyone at Thumbtack. They can browse and filter to find reports related to the question they have in mind or topic they’re interested in.

If you want to learn more about our research archive, check out this article for a behind-the-scenes look and an open-sourced version of the archive that you can copy and modify to suit your needs.

Finally, we’ve been bringing more cooks into the kitchen through interactive workshops and 1:1 coaching outlining the end-to-end process of running a research study. These workshops covered topics such as cognitive biases, different research methods, moderating best practices, and different ways to analyze data. We’ve also created a collection of guides and templates about how to use our tools, how to draft a study plan, and other team processes in order to make them easily replicable by non-researchers.

Icing

Nurture the growth of the research team.

Once this third cake layer has risen, it will be time to add the icing: bringing focus back to the researchers.

We’re not there yet, but our next efforts involve expanding the culture & growth of the research team. We plan to mix in things like professional development opportunities, adapting our onboarding practices to be successful in a remote-only world, and hiring more people to join the team.

While the opportunities for a research ops team are seemingly endless, our research ops team of one found success by first ensuring we had the essential components in place to make research happen. We then optimized our tool suite to gather and analyze the right data from the right participants, and finally, we made research accessible to our cross-functional partners.

We feel confident that we’ve invested in the right baking essentials for a one-person cake to stand tall, and hope you’ll take some time to think about what your bakery should invest in first!

___________________________________________________________________

This article was co-authored by Erik Olesund. Special thanks Lo Hayes for creating the illustrations, and Juliana Chang for editing.

Want to join the Thumbtack design team? We’re currently hiring for multiple roles. Check out our job listings here!