Beyond usability testing — Maturing a research team
New user research teams often start by focusing on usability testing — evaluating existing products and identifying aspects where users don’t understand how it works, or are unable to use it as expected.
The benefit of usability testing is immediately obvious — it highlights problems, which can be sized, prioritised and added to a backlog alongside more functional bugs. Fixing these problems has a reasonably clear return on investment, as we will see. Usability issues are simple to understand, and addressing them is simple to fit in with the rest of the team’s existing development process.
Although fixing issues is a great start for a new research team, focusing only on usability is limiting the value that user research can have. In this post, we cover what else user research teams should do for their organisations, and explore how to overcome some of the barriers to doing more impactful work.
What does usability testing miss?
Usability testing requires a product to be built before it can be tested. Even testing prototypes requires some commitment to “this is what we’re thinking of making” before deciding to test it.
This misses opportunities for inspiring radical innovation, and costs companies money.
Deciding ‘this is what we’re building’ before running studies means that companies don’t fully understand who their users are. They also won’t understand what issues users have, or where the business opportunities are. Understanding users better can help prioritise ‘what is the best problem for our product to solve’, and inspire creative ways of addressing the issues.
When the initial idea is wrong, and this isn’t discovered until later in development, changes need to be made to what is being built. Making changes has a financial and political cost, which costs companies money.
As the above chart demonstrates, It’s much cheaper to change ideas before production has started. It’s more expensive once development starts and some constraints get locked in. It’s almost impossible to make some significant changes to some product after launch, without throwing out a lot of very expensive development work.
Waiting until usability testing means discovering too late that the product isn’t solving a real problem, or isn’t addressing the top issues users have. This has a direct impact on adoption and the success of the product.
Why is there resistance to running research earlier?
Despite these benefits of running research studies earlier in development, it can be challenging to convince businesses to fund running ‘generative’ research before product development begins.
The benefits of user research studies are hard to quantify
There’s a few reasons why stakeholders can be reluctant. The first is that the benefit of these studies can be hard to prove. Unlike usability studies, it’s difficult to A/B test ‘what if we didn’t know this about users’, and so the financial benefit is hard to put into numbers. People quickly forget that they didn’t previously know these things about their users, and decide that the findings were obvious, without recognising how the study has influenced their decision making. Hindsight bias explains this — it’s extremely common for people to misremember the past and memories become distorted by new information when recalled. Researchers need to be aware of this risk so that they can generate genuinely valuable research objectives collaboratively with their colleagues, and document them at the start of their studies.
However before researchers get to kick-off to make the case for these advanced studies, researchers need to convince their colleagues research studies are financially justifiable. We will cover some methods to quantify the value of research shortly.
User researchers don’t recognise what is important to their colleagues
User researchers also sometimes don’t understand the incentives that their colleagues have influencing them, and how this impacts the language they use. Talk of ‘empathy’ for users is very common in the design community, but doesn’t seem very relevant when the stakeholder’s target is to double sales. Taking time to understand how the business works, what’s important to our colleagues, and building advocates is a powerful technique to overcome this — as we will see.
User research feels like just more opinions
Lastly, when misrepresented, these research studies can also feel like ‘just another opinion’ about what we should do. Everyone has opinions, so why waste time and money on discovering some more?
All of these points need addressing for research leaders aiming to raise the maturity of research in their organisation.
Changing minds about research
The first step for a research team should be to identify who makes product decisions inside your company. Then find what incentives influence their decisions. This can be treated as a research project, identifying how the business works, who influences decisions and interviewing each of them to uncover their goals, and the problems they have. This will help inspire the language a research team uses. It will also guide how to contextualise and disseminate the findings of research studies to make sure that studies feel relevant to the decision makers.
Make it clear research isn’t opinions, by separating ‘what we learned’ from ‘what we should do’
Helping people who are new to user research understand its role is essential. Be explicit about the difference between gathering information, and making decisions.
Research studies provide information that improves the quality of decision making. It doesn’t replace decision making, because business decisions involve a number of factors (technical feasibility, deadlines, achieving business goals) that go beyond what user research will uncover. Make this distinction obvious in the research process, by separating debriefing from ‘what we learned’ with any creative activities around ‘what should we do about it’. This will help raise people’s understanding of the role of research, and that its output informs opinions, rather than adding another opinion.
Gradually opening up the research process will achieve this. Creating opportunities for stakeholders to observe research, and take part in curated analysis activities, will help increase their literacy with research and evidence based decision making. This direct exposure, combined with preciseness in how research is run and analysed, will make it obvious that it is not just a collection of opinions.
Will Mydellton’s has documented his experience of opening up the research process to the whole team, based on his time working with government. His post has a lot of practical tips on how to achieve this.
Make the financial case by calculating the ROI of studies
The return on investment for usability testing is reasonably simple to quantify. One way to achieve this is measuring the impact on revenue when issues are fixed. For example, by fixing a problem which stops 10% of users from completing their purchase journey, fixing it should increase sales by 10% — a clear financial benefit. The impact of other types of usability issues could be measured by looking at the drop in support requests and calculating the value of the support team’s time.
More advanced ‘generative’ studies are harder to quantify. Since they inform long term decision making, and they don’t lend themselves to being A/B tested and are harder to give a value too.
One way to do this is by looking at previous research comparing the performance of companies who follow a full ‘design’ process, including research to understand their users. The design management institute tracked the performance of design-led organisations compared to equivalent organisations. In their 2015 report, they identified that design-led companies outperformed competitors by 211%. A similar 2018 report from the management consultancy firm McKinsey & Company also showed that design led companies increased revenue and shareholder returns at nearly twice the rate compared to other companies. Sharing this success of other companies can help get the attention of c-suite colleagues.
Another approach to justifying the financial impact of research is looking at how other teams do this. Marketing also suffers the similar issue that there isn’t a direct cause:impact relationship for their work. One of the ways they approach justifying the ROI is looking for the change in the trajectory of sales for a time period after their campaign runs. As researchers, we know that this is a flawed measure. It attributes all changes to the marketing campaign, whereas bug fixing, new features, or changes in the world will also be impacting sales. However being aware that this is how other teams are presenting their ROI will allow researchers to consider similar measures when required.
The Nielsen Norman Group have a case study looking at how quantifying the value of fixing usability issues had a big impact on Mozilla’s support website, where running iterative research studies helped reduce the support requests by 70%, helping them achieve their business KPIs and financial goals.
Find and develop advocates from successful projects
Another tactic for helping businesses recognise the value of user research is creating an environment where everyone understands the ‘point’ of research, and expects it as part of any project.
One aspect of achieving this is marketing the team internally. Posters, as used successfully by the UK’s Government Digital Service(GDS), can help promote the values that user researchers advocate, such as helping colleagues understand how different they are from their users, and how decisions are easier with data.
Combining this with education sessions, such as presentations, viewing research, and running interactive workshops to explain research will increase the organisation’s literacy with research. This makes conversations around running more advanced studies easier.
Running smaller, less high profile projects can help develop advocates. Often internal tools or initiatives get little attention, and are less glamorous, but are just as suitable for running research studies on. By starting off with these internal projects, a new research team starts to develop allies who have first hand experience of the benefit of running more advanced research studies, and advocate for these in internal case studies.
Moving beyond usability testing
As a research team matures, there’s a lot of ways in which their practice can develop. Opening up the research process with collaborative multidisciplinary activities will help grow enthusiasm for research. This will increase the impact a user research team has, and creates opportunities to influence projects earlier. Moving up the funnel beyond shallow ‘did we make it right’ work, to influencing decisions about what the company should make or do. My book ‘Building User Research Teams’ covers a lot of the practical steps required to establish and grow a user research team long term.
The strongest commitment an organisation can make to evidence based decision making is setting standards for projects. Again GDS have shown a model for how this can work, with each project being formally evaluated at gates such as the end of alpha and prior to launching. At each stage evidence is required that they have run appropriate user research, and this is assessed by experts. When empowered with real consequences, such as control over the budget, these project reviews can be a powerful tool for encouraging teams to run appropriate user research studies. However this requires a degree of faith which a user research team has to earn through good work and showing value — some of the tactics covered above will help with this.
Steve Bromley is an experienced user researcher. He works with companies to help them establish and grow new user research teams, and is currently leading a new research team for the UK’s largest publisher. He wrote about how to start new user research teams in the book Building User Research Teams.
Join the Leading Research community by signing up.
Follow us on Twitter @LeadingResearch for updates and articles.
