What to Do When Research Goes Wrong

Despite our best efforts, sometimes a project just fails. Here’s how to recover if it happens to you.

Mackenzie Greenwell

Published in

Meta Research

7 min readSep 8, 2021

By Mackenzie Greenwell, Dale Ramsey and Natasha Thalla

You planned what you thought would be impactful research. You thought through your methods and participants, and knew exactly how you’d analyze the data. You got buy-in and even some excitement from team members. Then you started executing your research project… and things didn’t go as expected.

As the three of us have learned firsthand, research projects can go wrong in a variety of ways. Maybe your survey doesn’t generate enough responses. Maybe your unmoderated interviews don’t reveal the rich insights you designed the study to produce. Or maybe your incidence rate and sample feasibility estimates were off, and now you can’t collect the survey data you need.

To be clear, we’re not talking about unexpected findings. Insights that challenge our thinking and raise new issues are wonderful and valuable, even if they’re difficult to parse. We’re talking about research projects that fail to deliver useful insights of any kind.

Despite our best efforts, these things happen. Trial and error is a natural part of the scientific process. The question is, how can we recover and learn from our “fails” — and help one another in the process? Here are some simple steps for evaluating, learning from, and pivoting after a project gone wrong.

1. Recognize the problem

Your ability to recognize when something is wrong with ongoing research can vary depending on method. With a survey, you may know there are complications brewing when data doesn’t start pouring (or even trickling) in, a day or two after distribution. With unmoderated testing, you may not notice issues until you’re analyzing the data and finding that it just isn’t as strong as expected.

After discovering an issue, consider how you might salvage at least some value from your research without totally depleting your sample population or going through the logistics and setup of restarting the project. You may be able to pull out a few interesting insights from your less-than-stellar unmoderated testing data, even if it only answers part of one research question instead of clarifying all the questions you originally set out to answer.

2. Reflect on root causes

When a research project fails, it’s essential to reflect on why it did. Maybe you felt pressure to move quickly to answer your team’s questions. Maybe you were using a new research method, getting feedback from an unfamiliar population, or digging into a totally new topic area. Maybe you dove into the project in order to make sure research was included as part of the decision making process. Reflecting on the contributing factors can help you better acknowledge the problem, avoid repeating the same mistakes, and evaluate your options going forward.

3. Find the best path forward

Sometimes you realize that what you’d originally set out to accomplish isn’t possible with a research toolkit (e.g., survey, interviews, secondary data). If research won’t cut it, partner with team members to identify potential alternatives for getting the answers to your original research questions.

If you determine that research is still an appropriate way forward, make sure that answering the original research questions is still a priority for team members. If not, it’s OK to recommend letting the project go. Otherwise, it’s time to evaluate your options. A quick cost-benefit analysis that takes into account timing, product roadmaps, your bandwidth, and possible re-prioritization can help you decide on an appropriate path forward — or lead you to conclude that there’s no saving this research effort.

4. Prepare to pivot

Before shifting to a new plan, be clear about the intended benefits of the changes you want to make. For example, will pivoting from a survey to interviews help you get richer data? Will it allow you to answer additional relevant questions that have come up since you began your initial project?

For us, developing a new plan usually takes about one and a half to two weeks. We affectionately refer to this as our “pivot plan”. Most of this time will likely be spent updating your collaborators on the need to pivot, assessing your own bandwidth, reprioritizing other projects as needed, evaluating other opportunities to answer your questions (with the help of managers, other researchers, and relevant stakeholders), clearly and concisely drafting a plan to share with collaborators, and going through any logistical or organizational processes needed to implement your new plan.

If it sounds like you have your work cut out for you, well, you do. But below we’ll share some helpful tips to get you started!

Write down your answers to the following questions in a new document — this will become your pivot plan — and will help you organize and clarify your thoughts, guide conversations with your team, and quickly bring everyone into realignment.

What background information will help my team?

First, you’ll want to include information about the questions you intended to answer with your initial research project and the method you used. Also share any successful parts of the original effort, like high-level learnings or new questions that were raised. Next, succinctly describe the specific areas that were problematic (e.g., data collection) and what went wrong (e.g., low response rates, unanticipated low-quality data). Describe why and how you made the decision to change course from your initial project and, if possible, how this pivot in particular will benefit the team. Highlight what you’ve done to remedy the problems of the initial project.

What are the remaining open questions?

Outline the research questions that still need to be answered, including any new questions. Consider whether the pivot might give you a chance to include questions that were cut from the original project.

What are our options?

Lay out the options you’ve identified for how best to pivot. To help set expectations and spark conversations with your team, outline the expected timelines, any relevant data explorations, and pros and cons for each option. Don’t forget to share your expert perspective about your preferred option.

What’s a feasible and appropriate timeline?

Present a realistic timeline for your pivot plan that takes into account product roadmaps, ship dates, etc. Help your stakeholders visualize your new timeline by building out a calendar or table with approximate dates for your new project milestones.

5. Reset expectations

Team members may not be privy to the details of the issues that arose with your project, or of the full pivot you’ll help the team make. Using the pivot plan you’ve crafted by answering the questions above, take time to realign with your team. Providing this written context can be particularly useful when there are a few potential directions forward (e.g., attempting another go at unmoderated testing or conducting in-depth interviews).

Sometimes realigning can be tricky. You may worry that your team will doubt you or the value of this research project, but it’s important to be honest about what has happened and what comes next. Demonstrating rigor in your assessment of the issues that arose and your pivot plan can help protect your credibility and reaffirm your commitment to getting the insights your team needs.

6. Find the silver linings — and lean on your people

As researchers, we feel personally responsible for our studies from start to finish. So when one of our projects fails, it feels like we have failed — like we’ve let down our teams and research partners and wasted everyone’s time. But in retrospect, our fails have been some of our most valuable experiences.

Don’t overlook the victories that just about every failed research project contains. For instance, maybe research and engineering learned to create and successfully launch a complex survey. Maybe you learned how to leverage a new-to-you tool or method. More importantly, you’ve almost certainly grown as a researcher and gained the ability to help other researchers by sharing your missteps.

If your organization has formal tools or systems such as peer groups, mentors, or coaches, be sure to take advantage of them. When our projects have failed, we’ve also sought help (and, let’s be honest, consolation) from other researchers, and sometimes even non-researchers. Every time, everyone has been willing to help, share stories, pass along knowledge, learn from our missteps, and lift us up.

It can be tempting to run from our failures, but when we approach them with curiosity, acceptance, and teamwork, we can emerge from them stronger, both as researchers and as a community.