Fighting insight waste with research repositories

Jake Burghardt

Published in

Integrating Research

14 min readFeb 22, 2021

Research reports pool insights, some of which become waste, and some of which undergo planning effort to become part of product backlogs.

Overview: Tech organizations are acting like laboratories without collective notebooks, unlocking only limited value from their diverse customer-focused research investments. We’re at local maximums for research utilization, and we need to grow our expectations. I spent 6 years tackling this problem at Amazon, and I’m continuing to investigate programmatic methods for collating research and activating it in planning processes. This post kicks off a new “Integrating Research” Medium blog to share some ideas.

Laboratories without collective notebooks

Research research everywhere — but is it actually used when plans are made?

In tech organizations, product teams continue to grow their focus on learning about people and how well they are serving them. From foundational research studies and market analyses, to dashboards tracking logged customer behaviors in products, to iterative evaluation of new and improved user experiences, to incremental A/B experiments, and more.

So many different types of research conducted. Quarter after quarter, and year after year.

But let’s be honest. These applied insights are often only momentary sparks, experienced by small audiences. Given daily pressures and cognitive load, many product teams have effectively treated insights from customer-centered research as fleeting, even when these insights could become durable through some additional care.

So even as more organizations are seeing themselves as being grounded in research, those same organizations have been operating as laboratories without collective notebooks.

Folks from various companies have told me the same story: product teams are unaware of what others within their ‘walls’ had previously learned. And the problem only grows as organizations scale.

The results? Well, there’s collective amnesia in product teams focusing on the ‘right now’ of the next new thing. And there’s the wasteful ‘Groundhog Day’ of repeated investigation of the same research topics. And most importantly, there’s decreased advocacy for what really matters to the people an organization is striving to serve. In the great lists of potential features to build, core customer priorities are not effectively represented at the top of the queue.

There are many underlying causes of under-utilized research outputs, such as unanalyzed data or the timing of insight delivery (topics for future posts). But to dig into what one area of research waste, let’s consider product teams’ responses to the essential attributes of a research output, such as the sample size or the content of individual insights.

Picturing an example output that teams might react to, let’s consider a broad-scope, exploratory research initiative. The kind of effort that takes a step back from myopic, feature-focused topics to instead investigate customers’ own worlds, employing a combination of direct observation, interviewing, and surveying. The kind of initiative that has the involvement of interested leaders, as well as solid collaboration with relevant team members.

Stakeholder responses to insights, ranging from FURTHER to CLOSER to planning use: ‘That’s out of scope,’ ‘That doesn’t sound right,’ ‘I’ll need to see more data,’ ‘That’s useful groundwork,’ ‘Yes, let’s do it.’

Even the top-tier insights from this kind of exploratory report can end up receiving very different responses, each with a pathway to the proverbial waste bin:

‘Yes, let’s do it’
These insights and recommendations appear to be on their way to immediate impact. They are clear problems to solve, often with clear solutions, at a scale that is amenable to product teams’ current positions in their timelines and processes.
These insights can become waste when… actions are agreed upon but not captured, prioritized, or implemented. Simply lost in the shuffle of many other ‘yes, let’s do it’ items.
‘That’s useful groundwork’
These insights help teams build empathy for the people they are striving to serve. Their primary value is shared understanding that clarifies purpose and provides meaningful framing for product efforts, rather than sparking specific ideas for features or improvements.
These insights can become waste when… relevant team members (or new people who have since joined) are not connected to the research, and they do not develop the same groundwork as their peers.
‘I’ll need to see more data’
These insights have made an impression, but may have a winding path to impact given current goings on. They may eventually land after other investigations uncover the same customer need and relevant product owners start to develop notions of what resulting solutions might look like.
These insights can become waste when… a small number of customer data points are unable to shift priorities and the topic is not actually revisited to build a stronger case for change [1].
‘That doesn’t sound right’
These insights conflict with stakeholders’ core assumptions or otherwise don’t seem important given current milestones and adjacent, overwhelming streams of incoming information.
These insights can become waste when… leaders persist with their own points of view in the face of contradictory evidence — evidence which eventually becomes lost in collective amnesia.
‘That’s out of scope’
These insights are simply too far outside of what most folks in an organization would consider owning, and product leaders can’t quite picture what solutions might look like. Insights in this category can be unmet customer needs, within the scope of an organization’s vision, that leaders don’t see as a priority — but which may turn out to be an entry point for competitors.
These insights can become waste when… they are not recalled during strategic planning, outshined by a focus on existing feature areas.

These five responses are far from an exhaustive list of how teams may react to a research output. But I hope these examples help paint a picture of how research waste, in the form of under-utilized insights, could accumulate within an organization over time.

Maybe all those insights will be reconsidered for the next version? Next release? Next year? Not on their own they won’t.

Being honest about local maximums for research use

All of this arm waving about unapplied research is not to say that there aren’t plenty of amazing research wins in today’s world. There absolutely are. And these wins have considerable impact on people’s everyday experiences and organizations’ ongoing successes.

I’m also not here to say that total use of every last bit of research is the goal. It’s a given that, due to limited resources, product roadmaps and backlogs require what may seem like ‘ruthless’ prioritization. By necessity and product strategy, many insights won’t make the cut.

But today’s research utilization in many organizations is at a fraction of its potential, especially once product offerings are launched and being iteratively improved.

And by research utilization potential, I mean the capacity to fuel plans for better serving people, advancing organizations toward their missions and accomplishing their core goals.

Utilization of research in planning, from local maximums to bigger expecations.

Often, it takes leaders so much hard work to get to one peak of research utilization and real-world impact, that it can be hard to consider other, higher peaks. The question fueling my recent work has been: How can organizations develop bigger expectations of research value and start acting more like a laboratory with a collective notebook.

Setting bigger expectations for research use

Below are some questions that may be useful for growing expectations and considering what’s possible for research contributions and impact. For each question, you might consider what these outcomes could look like in your own organizational world, and what it might take to make progress in your particular landscape of teams, culture, design, and product management practices.

What if our existing research was consistently:

Still in use long after the initial readout?
Planned against broadly, across a range of findings, rather than just a couple key headlines?
Perceived as a trusted, growing body of knowledge, rather than a series of disconnected reports and dashboards?
Connected to each of the teams where specific insights could be valuable, regardless of which team originally sponsored the research?
Integral to each major effort in product teams’ own road maps?
Integral to how leaders shape their ideas about what they want their organization to tackle next?
Integral to organizational design and determining where staffing investments are made?

Sound like I’m dreaming the impossible dream? Well maybe, in part — but I’ve seen some successes within each of the above areas (as well as failed experiments to learn from). At their root, each success was enabled by some sort of research repository.

My journey using research repositories to fight insight waste

Who would have thought that research repositories — internal knowledge management and planning support tools focused on customer understanding — would become a hot topic? Well, ‘hot’ among researchers and designers of tech products at least, and ‘getting warmer’ for their collaborators.

Research repositories can be as basic as a centralized internal destination for research reports, or they can be something much more complex: deconstructing research outputs into their constituent elements and organizing those elements to better connect into product planning over time.

I, for one, would not have guessed that substantial investments in research repositories would become seen as something of a new best practice — and I spent 6 years activating research at Amazon, owning repositories of collated insights from multiple research sources, first in Retail and then in Alexa and Devices (learn more on my LinkedIn profile).

Why wouldn’t I have guessed it? Well maybe it’s because, when I got started on amplifying existing research done by colleagues rather than conducting my own research, I wasn’t aware of anyone in tech working in a similar vein [2]. Admittedly, I didn’t look hard enough and just got to work.

I was fueled by a nagging observation that traditional approaches to communicating research insights have not been delivering as much value in product planning as they could be, especially when considered over time. I wanted to work on reducing waste and improving impact — on making research the lifeblood of ongoing planning in product teams, not just an auxiliary contribution.

I wanted core customer insights that had slipped through the cracks from past research reports to stand up and push their way into today’s product roadmaps. Not as some sort of vindication for researchers everywhere, but as obvious customer and business value left on the cutting room floor.

I didn’t realize that this inkling would set me on a new chapter in my career as a meta-researcher, standards definer, hack librarian, community manager, cross-silo-prioritizer, customer bullhorn, distributed ideation facilitator, experiment accountant, and much more. As people often do, sometimes I wonder “how did I get here?” [3]. But all and all, it’s been an interesting chapter. I get satisfaction out of supporting both researchers and insight seekers of different stripes as they work to make better products. I enjoy fighting the ‘unknown known’ and connecting the dots of the ‘customer needs forest’ from individual ‘research finding trees.’

It turns out, like most good ideas, a lot of other people had similar inklings that research could be better collated and activated at a granular level. It’s been exciting to see sharing on the topic of research repositories in tech organizations increase over the last few years [4]. It’s also been great to learn from the growing ResearchOps community, including its focus on repositories [5].

I’m taking a breather to focus on my family during this Covid-19 era, and I’m excited to take some time to dive into more of what’s out there regarding research repositories, insight operations, and voice-of-the-customer-driven product management.

‘Integrating Research’ table of contents

As part of that diving in, I’m planning on doing a bit of Medium blogging, with the goal of rolling it up into a book. If you’ve read this far, you might be interested in the list of draft topics that have started to appear at my keyboard.

2023 UPDATE: This table of contents has pivoted into ‘Stop Wasting Research’ book for Rosenfeld Media > Subscribe for updates

Integrating Research: New insight operations to increase impact and deliver products that matter. Presents a visual of the table of contents that follows, below.

EXPANDING REACH AND CONNECTIVITY

A. Envisioning existing research at the center of product innovation

A1. [This article] Fighting insight waste with research repositories
A2. Aiming for integrated research, not just a research repository tool
A3. Extending insight ‘shelf life’ to get more value from research in product planning
A4. [On Re+Ops Community Medium] Opening the gates: Addressing researchers’ concerns about broadening access to research repositories
A5. [On Re+Ops Community Medium] Activating insights to overcome common barriers to product impact
A6. Mapping research impact in product development
A7. Imagining new architectures for connected research repositories and systems
A8. Using research knowledge to balance product leadership and management mindsets

B. Turning up research communications

C. Normalizing research reuse to lay the groundwork for new operations

FORGING SYSTEMIC KNOWLEDGE

D. Improving upon existing research knowledge management practices

D1. Enabling collaboration with standards for describing research across teams and disciplines — to get more done with insights
D2. Connecting existing research repositories owned by different disciplines and teams — to get more done with insights
D3. Auditing governance in existing research repositories and applying new standards — to get more done with insights
D4. Reconsidering success for existing research repositories and applying new standards — to get more done with insights

E. Establishing new tooling, systems, and programs to enhance research knowledge management

E1. Envisioning new knowledge management successes and outcomes — to get more done with insights
E2. Designing research system architectures and programs to achieve new knowledge management outcomes — and get more done with insights
E3. Growing buy-in for what new research repository programs will require, capture, store, and activate — to get more done with insights
E4. Driving research findability through primary research repository pivots and essential metadata — to get more done with insights
E5. Cultivating the growth of high quality insights in research repositories — to get more done with insights
E6. Pushing cross-study reporting from research repositories to build mindshare for crucial learning — and get more done with insights

F. Managing research in repositories as learning accrues and ages

F1. Clarifying related insights into a consensus view within a research repository — to get more done with insights
F2. Structuring emergent insight themes in research repositories to show the ‘forest’ and the ‘trees’ — and get more done with insights
F3. Ranking insights in research repositories to surface what matters most — and get more done with insights
F4. Removing expired evidence from research repositories — to get more done with insights
F5. Archiving insights that are no longer applicable in research repositories — to get more done with insights
F6. Examining gaps in research repositories to inform research study roadmaps — and get more done with insights

CRAFTING ARTERIES INTO PLANNING

G. Driving early adoption of repositories in product design processes

H. Onboarding all product people to repositories and their culture of research use

H1. Pushing for citations of research repositories in product practice — to get more done with insights
H2. Starting with related research while identifying product questions — to get more done with insights
H3. Training product people on how to find and apply available research — to get more done with insights
H4. Amplifying the emergent practices of research repository ‘lead users’ — to get more done with insights

I. Expanding the integration of repositories into ongoing product planning

I1. Using research repositories to connect specific insights to more of the ‘right’ product teams — to get more done with insights
I2. Revisiting aggregated research when product leaders are actually thinking big about planning — to get more done with insights
I3. Integrating existing research into product teams’ ongoing backlog, prioritization, and road map process — to get more done with insights
I4. Tying research repositories into goal setting and reporting that product leadership are held accountable to — to get more done with insights

J. Tracking research progress toward product impact

J1. Tracking feature and service launches that were initiated based on research learning — to get more done with insights
J3. Tracking product team touch points with research repository contents — to get more done with insights
J2. Tracking the ‘distance’ of individual insights from product release — to get more done with insights
J4. Demonstrating individual researcher, team, and community impact in research repositories — to get more done with insights

CATALYZING PRODUCT CHANGE

K. Advocating for shifts in product direction based on accumulated research

K1. Challenging product teams’ ‘tunnel vision’ assumptions using research repositories — to get more done with insights
K2. Spotlighting crucial learning in research repositories to advocate for new vectors in product planning — and get more done with insights
K3. Alerting product leadership to harm and conflicting values documented in research repositories — to get more done with insights

L. Retooling product organizations based on accumulated research

L1. Institutionalizing new product success metrics based on learning in research repositories — to get more done with insights
L2. Identifying problematic root causes in product development based on learning in research repositories — to get more done with insights
L3. Driving new product teams and other organizational changes based on learning in research repositories — to get more done with insights

Maybe none of this is news to you. Maybe you’ve got all of your researchers’ insights from various disciplines fully integrated into your organization’s iterative and long-term planning processes. (If so, please let the world know how you did it — especially if you’re at a mid- or large-sized organization working on an already-launched product.)

However, if any of the above piques your interest, please don’t be a stranger. I’m curious to hear about your challenges and successes reducing research waste. Thank you!

Connect on LinkedIn
Sign up for email updates (monthly, at most)

—

Footnotes:

[1] A favorite summation of a common problem: “We get used to hearing ‘we will come back and fix it later,’ and then later never comes. In two weeks, it will be on to the next problem and the next feature.”
2020, Jarvis Moore: “Why Don’t We Talk About UX Debt?”
https://dscout.com/people-nerds/ux-debt

[2] I started working full-time on insight reuse and insight repositories to inform product requirements in 2013. Applied research repositories of various types were established in other industries long before then, and report repositories are nothing especially new in tech organizations. I’m sure there was existing published work that my colleagues and I overlooked and should have built upon — any pointers to older foundations for insight repositories in the tech industry would be greatly appreciated! I’ll post what I surface as well.

[3] I didn’t realize until after the fact, after years of working on insight repositories, that I had been manifesting a vision from one of my professors at the University of Washington: Judy Ramey. In a usability testing class in 1999, Judy had taken to the chalkboard to sketch a system of rationale that traced product requirements back to a repository of video clips from research sessions.

[4] Selected examples of research repository articles that first provided a larger industry context:

2016, Tomer Sharon: “The atomic unit of a research insight” https://medium.com/@tsharon/the-atomic-unit-of-a-research-insight-7bf13ec8fabe
2017, Louis Rosenfeld: “Beyond User Research” https://www.slideshare.net/lrosenfeld/beyond-user-research/
2018, Kate Towsey: “A framework for #WhatisResearchOps” https://medium.com/researchops-community/a-framework-for-whatisresearchops-e862315ab70d
2019, Matt Duignan: “How Microsoft’s Human Insights Library Creates a Living Body of Knowledge” https://medium.com/microsoft-design/how-microsofts-human-insights-library-creates-a-living-body-of-knowledge-fff54e53f5ec
2019, Etienne Fang: “The Power of Insights: A behind-the-scenes look at the new insights platform at Uber” https://medium.com/uber-design/the-power-of-insights-a-behind-the-scenes-look-at-the-new-insights-platform-at-uber-26f85becc2e6

[5] Very thankful to all of the #ResearchOps folks, who are building professional community and sharing a curated feed of valuable content: https://medium.com/researchops-community/