Discover Your Strategy: How to Craft Research That Matters

There is a widespread misconception when it comes to market research: that data takes the risk out of decision-making. We believe that by supporting insights and strategic decisions with hard data we are able to prove that we’re doing the right thing. But that little asterisk at the bottom of the report doesn’t just say the sourcing. It implies a certain method of arriving at conclusions and all the assumptions built into it. And no, a big sample size won’t save you.

Every methodology carries a risk in the assumptions it requires you to make, and no methodology is without assumptions. What’s more, every researcher, manager, and CEO carries confirmation bias, and no human is without psychological biases. The result is our current landscape of wasted time and resources, misleading findings, and a general disregard of research in decision-making.

But what if there were a way to acknowledge and compensate for risk in research? At Theory we use what I have termed Analytical Hedging, the practice of compensating for research assumptions by combining an array of methods. By hedging against the risks in research and dividing up our budget into a series of light-touch methods we achieve a faster, more accurate, and lower cost output. The result is a focused, usable truth that is made to inform the strategic decision at hand, not to sell more research.

The Hidden Risk of Research

There is no such thing as a singular iron-clad research method. Every approach comes with a series of assumptions that, unless acknowledged, can add risk to the decision we’re trying to make with this research.

Assumptions aren’t always a bad thing. In fact, they provide necessary clarity in a complex world, allowing us to evaluate our hypotheses and observations. The signal without the noise or whatever. But in academic social science, we start with a simple model with many assumptions built in, and then compensate our analysis order to control for added complexity. There is real usefulness in examining our world through the lens of a simplified model, but we must remember that whenever we add in assumptions we are designing research for a simple world, not the real world.

We need to stop kidding ourselves about the soundness of our research. Data collection, interpretation, and presentation are intrinsically flawed with assumptions and biases.

There is risk whenever we are given a research task: sample bias, confirmation bias, client-pleasing bias. Our brains are wired to detect patterns where there are none. There is risk in ethnography, where the world of a few may not represent the world at large. There is risk in experiments, where your controlled setting may not mirror the real-world environment. There is risk in focus groups, where groupthink, loud voices, and bandwagonism can sway a conversation. And there is risk in secondary research, taking on all of the original researchers’ flaws plus the bias of your own agenda.

There is especially high risk in surveys, where the presence of numbers pacifies stakeholders into accepting statistically insignificant mean comparisons. Selection bias aside, even the most minute subtleties of wording and format can make all the difference in survey outcomes. A recent study indicated that there are 48 types of common biases, including seemingly trivial choices as whether or not survey selection buttons are on the left- or right-side of the question. Here’s an example from Pew on how question wording can significantly change survey question outcomes:

Not only is there risk in methodological assumptions, but also in the competency and thoroughness of the researchers themselves. And this risk is growing: with the commodification of surveys and cheap access to secondary data, the gap between research and well-informed decision-making has only widened.

Using Analytical Hedging

Research is an inherently valuable yet risky asset, and the best approach to overcoming this risk to compensate for it with a mix of methodologies. Even academics (arguably the highest echelon of research) is peer reviewed and updated as a way to acknowledge risk. Some call this approach Abductive Inference or Meta-Analysis; I call it Analytical Hedging. By intelligently combining findings from four, five, or six research methods we can design around assumptions and find a definitive truth. On top of that, Analytical Hedging allows researchers to have a light touch on each method, because findings will be generated across a large number of efforts.

Think of it like a financial portfolio: you wouldn’t want to have all of your savings invested in one stock. And sure, the company might be successful now, but there is still the inherent risk of volatility when placing just one bet — better to diversify your assets into a smart mix of different types of assets (stocks, bonds, T-bills). This principle also applies to research: do not rely on one methodology, even if the sample size is huge.

Adding more methodologies has a narrowing effect, transforming research findings and data into focused, usable truths.

In contrast he output from one comprehensive survey, for example, can result in many cuts of data and unfocused findings. But both author and analyst are blind to their own biases, and by nature cannot tell when they are distorting information. When we add more methods we are empowered to leverage and reinforce the commonalities between studies, and also to disregard misleading information.

In practice Analytical Hedging is specific to the resources available, and we would select from the following approaches depending on the nature of the decision at hand. We typically start with a review of public resources, including financial statements, press releases, search trends, and experience audits. This is combined with observational work, including on-site field work, platform data & analysis, and expert interviews. Finally, we would also want to include experiments such as surveys, in-market A/B testing, and concept testing workshops. With a combination of some or all of the aforementioned, the common threads begin to connect around market truths. The final product should be a document centered around these truths or decision points, with facts from multiple sources as proof.

Tangible Success

My team recently took on a marketing project for a design museum, planning member programs and a new brand positioning aimed at fundraising. Using stakeholder interviews and ethnography, we found that there were clear opportunities to enhance the museum’s membership program through enhanced benefits and on-site messaging.

But our analysis did not end there: using financial statement analysis we discovered that the membership program for this organization had a low and decreasing ROI year over year. And this finding completely changed the trajectory of our recommendations. By hedging our research efforts, we were able to pivot our strategy from targeting membership to targeting potential board members and volunteers. Not only did we hone a high-return strategy, but we also successfully avoided going down the wrong path in implementation.

Our goal well-informed decision-making. Sometimes it feels like clients and stakeholders try to question methodologies because they don’t agree with the findings. They’re not wrong, though. It’s time to cull our over-reliance on surveys and analytics, which provide a false sense of security. Using just one method isn’t enough to declare a solution without the risk of assumptions. Analytical Hedging, a combination of light-touch research methods, is low-cost and high-value because it provides truth without risk.

Research stakeholders ultimately want to make a reliable decision, and therefore research should be valued on its usefulness, not its sample size. It’s no wonder so much research sits on company shelves collecting dust.

Andy Rooks is the founder and strategy director for Theory Marketing Partners, a selective consultancy specializing in research and strategy services. Find out more at www.theorymarketingpartners.com

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by 355,974+ people.

Subscribe to receive our top stories here.