Let’s talk research impact
In November last year Christina Li & I started this initiative on Leading Research. We decided to do this after many discussions about how lead researchers could support and learn more from each other. We are super excited to kick-off this initiative by discussing with you all the much debated topic of research impact.
We held an event in January 2020 at Monzo to uncover how leaders talk about research impact. Here are our learnings.
Research impact: the problem
There’s currently no easy way to measure research impact and one size does not fit all. People are often trying to quantify impact — because we’re so used to quantifying things so we can measure and benchmark them. However, not all aspects of user research can be measured in such a precise manner.
Why is it important to discuss research impact?
- We want businesses to value their investments in us. As leaders, we want to help increase the funding, growth and maturity of research teams.
- We want our impact known. As researchers, we don’t always get visibility of when we’re making impact. And, how we can communicate this impact to people higher up in the organisation.
- We want to be a strategic partner. Research is key in driving the strategic vision of a company and product — here’s an example of how research guides Airbnb product development. To deliver research at a strategic level, we need to communicate and show the impact of research. Strategic research takes more time and space, and therefore requires more buy-in from senior stakeholders.
So, the question we found ourselves asking was: How might we prove the impact of research? Especially considering the less visible aspects of research.
What we did: Roundtable with lead researchers
To dig deeper into this topic, we gathered 10 research leaders in the room. We held the event in a roundtable discussion format. We deliberately kept the event small. This was to encourage sharing of real challenges as well as successes.
We also provided everyone with some prompting questions before the event, so there was time to think through the challenges and prepare for the discussion.
Some of our questions
- What does ‘research impact’ mean to you and the organisation you’re working with?
- Why does ‘research impact’ matter?
- Are there stories you can share with us from your career where you showed impact? What were the lessons and challenges?
- Should impact be the same across different teams and levels in the organisation?
Key insights from our discussion
1) There are different levels of research impact
Cultural impact
Cultural impact is about building relationships and changing mindsets. This could have a slow start, but it can have a more lasting impact.
This is the Aha moment. It’s when someone learns something new through research. Often cultural impact is more informal and harder to measure — for example, a non-researcher colleague telling you after attending research, how they found the insights and the experience itself truly eye-opening, and how they now have a better understanding of what we do.
How to create cultural impact:
- Setting up a research insights exhibition in the office. Invite non-research colleagues so they see and benefit from the insights you gathered.
- Taking colleagues with you on contextual visits. This helps them observe and understand the users and they can discuss their perspective with you.
- Storytelling. It’s another valuable way to bring insights to life. It builds empathy and change mindsets.
Outcome-based impact
Outcome-based impact is about learning from the research done and responding to the actions. This type of impact can be hard to assess when researchers are not embedded in product teams, or when there are fewer researchers in the team to have oversight. Despite these challenges, it is important and helpful to see the types of decisions made and how research findings informed design. Keep track of changes that go live from research. And ‘measure’ the impact in real life.
2) Research impact depends on many factors
Research team size is important, but there’s no ‘magic number’
It’s not as simple as a bigger or smaller team, or a specific magic number. A big research team alone cannot help make an impact. It’s about balancing the team size to the rest of the organisation. If you want to be truly research driven, then it’s about ensuring that each researcher can allocate a suitable proportion of their time towards sharing the insights and guiding decision making (in addition to planning and doing research). If a researcher is spread out too thinly, they cannot be in the room to guide decisions based on the insights.
Priorities of a research team of one
If there’s just one initial researcher and the company is open to research, then it’s easy to make that big impact quickly. The trick is in picking projects that can prove strategic impact quickly. If you’re the first researcher in the company then anything you do is going to be eye opening and rewarding for the whole team. And then it will be the right time to bring more researchers in to do more.
The ratio of researchers to designers
There’s not always a balance of researchers and designers — for example, one researcher to every four designers, every 10 developers, or worse. Research teams still often sit within design teams. We often don’t have an equal position in businesses. This can mean researchers are not embedded in product teams / squads. So ultimately, researchers miss key moments in building closer relationships.
Emotional burden for researchers
It’s not all research all the time. To prove value of the work and ensure the research is used, we need to be sharing and evangelising too. A smaller team means there’s a lot of work for this team to share the value of research, communicate the insights and do the research (and don’t forget the research ops!).
3) Demonstrating impact depends on research maturity
Research is not just about design ‘validation’!
We shouldn’t just be doing evaluative research, or research that’s reactive to the needs of the product team. This is the most common form of research in product teams. Whilst evaluative research is valuable, this should be a standard part of the product development process. To show the value of research at a higher level in the organisation, we should be doing more generative and strategic research that can inform the roadmap.
How risky is the problem or solution?
We shouldn’t just be researching everything either. When you have a small research team, it’s important to prioritise the research we work on and the questions the team want answered.
How to prioritise research to deliver the highest impact:
- Pick research projects based on how uncertain the team is about the topic.
- Pick research projects based on how risky it is for a solution to go live without feedback from users.
Generative research value is harder to measure
With the rise of lean UX research, we’re often focussed on how research can fit in a weekly or two-weekly cycle. And we’ve optimised our analysis and synthesis too. This is great as it stops research from being a blocker. Yet, lean research is sometimes to the detriment of generative and longitudinal research. These fast-paced research approaches have made it harder to convince stakeholders of the value of spending time to know the users: For example, observing people in context or asking people to complete a diary over 2 weeks.
So what does this mean: can we measure research impact? And should we?
It’s not necessarily about putting an ROI to the value of research. Trying to ‘measure’ impact is a symptom of the underlying problem: research is not embedded in the product development process. We need to demonstrate the value without measuring impact as such, because ultimately research should be an integral part of the process.
Let’s make this our shared goal: How might we ensure different types of research are embedded in the process, at the right stages. And we’ve identified some ways of approaching this.
- Being proactive: Let’s have a shared roadmap of what we’re working on as a team. Research can be ahead and feed into this roadmap. We should have topic areas and problem focussed roadmaps rather than feature focussed, so that we can understand user needs better first (before solutionising).
- Creating a knowledge base: Let’s retain knowledge and figure out how best to save these insights. Some companies are now hiring people to do Ops and this involves knowledge management as well. How can we synthesise research so that it can tell us something useful?
- Communicating research insights regularly: For example, a Research day every quarter sharing insights, regular newsletters, reflecting and presenting back key insights and actions from the last year.
- Exposure hours: Encourage team members to join you in research sessions, and count how many non-researchers are getting “exposure hours” to research. They can count this towards their personal objectives too.
This is not an exhaustive list, and there’s more we could be doing. We’d love to hear from more researchers to know your thoughts and ways we can start sharing the amazing work that we’re doing.
We are making an impact, so let’s share it ☺
Acknowledgements
Special thanks to the researchers who participated. And to Samantha and Monzo for hosting us. We had a mix of lead and senior researchers from a range of industries, as follows:
- Anja Maerz (Lead Researcher at Babylon Health)
- Anne Stevens (Director of UX Research at Culture Trip)
- Audrey Chung (Senior UX Researcher at Deliveroo)
- Ben Garvey-Cubbon (Senior UX Researcher at OVO Energy)
- Katherine Vaughan (Senior User Researcher at Monzo)
- Lee McIvor (Product Director at Abcam)
- Liz Kessick (Director, Elizabeth Kessick Consulting)
- Samantha Davies (Senior User Research Manager at Monzo)
- Tricia Lee (Director, Tricia Lee Ltd)
The roundtable was planned and facilitated by Christina Li (Director of Melon Experience Design) and Swetha Sethu-Jones (me, Senior UX Researcher at Just Eat).
About Leading Research
Last year Christina Li & I started this initiative on Leading Research and thanks to this amazing community we’ve had nearly 40 senior and lead researchers from the UK sign up to support this initiative. If you’re a lead or senior research practitioner you can sign up.
