Five dysfunctions of ‘democratised’ research. Part 3 — Research as a weapon
This is the third in a series of posts examining some of the most common and most problematic problems we need to consider when looking to scale research in organisations. You can start with the first post in this series here.
Here are five common dysfunctions that we are contending with.
- Teams are incentivised to move quickly and ship, care less about reliable and valid research
- Researching within our silos leads to false positives
- Research as a weapon (validate or die)
- Quantitative fallacies
- Stunted capability
In this post, we’re looking at what happens when research is ‘weaponised’ in teams.
Dysfunction #3 — Research as a weapon (validate or die)
Over reliance on research, without care to the quality level of the research, can also be a symptom of another problem in our organisations — lack of trust between disciplines in a cross functional team.
In particular the relationship between design and product management can have a substantial impact on the way that research is used in product teams. If the relationship is strong, aligned and productive then research is often used to support real learning in team. But where the relationship is less healthy, it is not uncommon to see research emerge as a form of weaponry.
Winning wars with research
How does research become weaponry? When it is being used primarily for the purpose of winning the argument in the team.
Using research as evidence for decision making is good practice, but as we have observed in earlier dysfunctions, the framing of the research is crucial to ensuring that the evidence is reliable and valid. Research that is being done to ‘prove’ or ‘validate’ can often have the same risk of false positives that comes from the silo dysfunction.
This is because the research will often be too tightly focussed on the solution in question and there is little or no interest from the team around the broader context. This lack of realistic context can result in teams believing that solutions are more successful than they will ultimately turn out to be in the realistic context of use.
Data as a crutch for design communications
Another reason to see research being used as weaponry is to compensate for a lack of confidence or ability in discussing the design decisions that have been made. Jen Vandagriff, who I’m very fortunate to work with at Atlassian, refers to this as having a ‘Leaky Design Gut’.
Here we see research ‘data’ being used instead of (not as well as) the designer being able to explain why they have made the design decisions they have made. Much as I love research, it is foolish to believe that every design decision needs to be evidenced with primary research conducted specifically for this purpose. There is plenty we know about design decisions that make usability better or worse.
In a team where the designer is able to articulate the rationale and objectives for their design decisions, and there is trust and respect amongst team members, the need to ‘test and prove’ every decision is reduced.
Validation can stunt learning
Feeling the need to ‘prove’ every design decision quickly leads to a validation mindset — thinking, ‘I must demonstrate that what I am proposing is the right thing, the best thing. I must win arguments in my team with ‘data”.
Before going straight to research as validation’, it is worth considering whether supporting designers to grow on their ability to be more deliberate in how they make and communicate their design decisions could be a more efficient way to resolve this challenge.
Sometimes it is entirely the right thing to run research to help understand whether a proposed approach is successful or not. The challenge is to ensure that ‘validation’ and settling arguments doesn’t become the primary role of research in the team.
If we focus entirely on validation and ‘proof’, we risk moving away from a learning, discovery mindset. We over index on the leanest and apparently definitive practices. With this mindset, our research is often comparative (A/B testing) and our outputs are often reductive (eg. scorecards). We’re incentivised to ignore any flaws in the validity of the method if we’re able to generate data that proves our point.
Alignment over evidence
Often this behaviour comes from a good place. A place where teams are frustrated with constant wheel-spinning of everyone having an opinion about design. The team is trying to move away from opinion based decision making, where either the loudest voice always wins or the team feels frustrated by their inability to make decisions to move forward. Using research as a method to address these frustration does make sense and should be encouraged.
Validation research can seem to provide short term results to help move teams forward, but it can also reinforce a combative relationship between designers and product managers. Often this relationship comes from a lack of alignment around the real problems that the team are setting out to solve. Investing more in more discovery research, done collaboratively as a ‘team sport’ can be very powerful in helping create a shared purpose across the team that can help promote a more constructive and supporting teamwork environment.
Support from an experienced researcher with sufficient seniority can help the team avoid the common pitfalls of seeking the fastest and most definitive ‘result’, but to achieve a shared understanding of both the problem and the preferred solution. Here the practice of research, done collaboratively as a team, can help not only to inform the situation to achieve more confident decision making, but also to heal some tensions in the team, by bringing the team together around a shared purpose — solving real problems for their customers or users.
First published on www.disambiguity.com