Doing product impact analysis as a team: practical guidance
Many of us know the mantra: Build the right thing, and build the thing right.
As product people, we do this already by thinking about user needs, testing our assumptions, continually improving usability. You can extend this by also thinking about the ethical impact of a product.
Our team, Wellcome Data Labs, has been tinkering with product impact analysis for more than a year. ‘Product impact analysis’ is a fancy (and more concise) way of saying: ‘understand if we are building a thing that won’t harm people and has some positive impact in a fair way’.
We’ve drawn on tools from human-centred design, DotEveryone’s Tech Transformed, and product management to find a way that works for us.
I’ve facilitated this analysis through a couple of projects and products in Wellcome, and each time was different. A lot depends on who is involved and where the product is currently at. This blog post outlines how our teams in Wellcome Data Labs has gone through process.
Why do impact analysis?
The goal of impact analysis is to:
1) anticipate some negative outcomes that your technology may contribute to.
2) to have a strategy to minimise this risk.
Who to involve?
This exercise is aimed at people who create and build the technology. Typically this would include the project/product team and any close stakeholders.
To avoid blind-spots, you should also include voices that represent the domain in which you are creating a technology solution. In Wellcome Data Labs, we run an internal working group that includes voices from different departments throughout Wellcome. This working group participated in one of our early impact analysis workshops.
What is involved? Consequences and uses
DotEveryone’s Consequence Scanning has been really useful. By drawing out the intended and unintended consequences of a product/project, it has really helped our teams know what we are concerned and not concerned about.
As well as consequences, we also have focused on intended and unintended use. If you aren’t 100% clear on how you want your tech to be used then anticipating the consequences will be harder.
Below is an outline of the different areas that are useful to form team consensus and clarity on. I am going to use the example of a simple commuter bike to illustrate these.
Planning and running the analysis — example of the commuter bike
1. Function
What does the product/service/tool/ thing do?
This may be written down somewhere in your team documents or you may need to work this out with the manager of the team. This is important as it forms the basis for all other analyses.
Bike example: The commuter bicycle is a form of transportation. By peddling the wheels, the rider can travel of speeds up 15km per hour on roads and pavements.
2. The intended consequences
What does the team want as a positive knock-on effect as a result of their product?
A set of intended consequences are like a vision statement, i.e. what will change as a result of the tech creation? It’s best to go as far-reaching as possible, thinking of how the product will impact society in the long run. It should link back to your organisation’s mission.
Bike example: The commuter bike will allow people to travel conveniently and freely, as an alternative to driving or public transport, therefore reducing reliance on fossil fuels and contributing to the fight against climate change.
Here’s an example in the Wellcome Data Labs context:
We have built a tool, called Reach, that is able to find where research publications have been cited in policy-related publications. We believe that this will help surface how research is influencing policy. The intended consequence is that by surfacing how research is influencing policy, we encourage evidence-based health policy, which in turn supports Wellcome’s mission to ‘improve health by helping great ideas to thrive’.
There can be a false sense of consensus within a team on the intended consequences of a product or project, sometimes because it is seen as too high-level. In the Reach team, we started out with a direct connection with Wellcome’s mission when it was an internal product, but when we sought to make it public we sometimes struggled to make decisions when there were divergent intended consequences within the team. We didn’t quite make the connection to what impact we would like to have on society.
How you draw this out from a team depends on the team’s structure. In previous efforts I have found this needs ironing out with the manager and then agreement from the team. To facilitate this part, be sure to use a technique such as 1–2–4-all to draw consensus from the bottom-up and in an inclusive way.
3. Use cases
How does the product team intend for people to use it?
Think of what the product was designed for and who the product team designed it for. This is different from stating the function — it’s about examples of use.
You can draw out the above use cases in advance by speaking to the UX designer and the product/project manager (if you aren’t this person already!). Then you take these as the starting point for filling out the ‘use case quadrant’.
Misuse: UX designers will note that the misuse case is what happens when the conceptual model does not match the user’s mental model: they end up using it in a way that you, as the designer, did intend for them to use it, and the user doesn’t know they are misusing.
Abuse: The abuse case has more relevance to the security and risk analysts within the team, as well as having an ethical concern.
Stress: Stress cases are interesting, less concrete. The best explanation I have of them is from the work of Eric Meyer & Sara Wachter-Boettcher in their book Design for Real Life. The stress cases are the step before an unintended consequence — it’s a usage that is correct from the perspective of the product creator, but results in causing stress or harm.
Facilitation techniques
Here’s are some ways I have facilitated the use case analysis in the past:
Using the 1–2–4-all technique from Liberating Structures (or a variation) we work through the use cases first (if not already worked out with designers and PMs in advance), and then on to abuse, misuse and stress.
There are a couple of ways this you can do this:
- For each part of the quadrant people write their thoughts on post-its, share in 2, then 4s (if group is big enough) and then share with everyone. By having a group discussion for each four parts of the quadrant, you can get into more detail and be more focused.
- Alternatively, the individuals can think of all the cases all at once, share in pairs, in 4s, and then give feedback to the whole group. This could help the conversation be more exploratory as people can see connections between the cases as they go through each one, but the outputs are a little harder to process as a group.
4. The unintended consequences
Function, intended consequences, and use cases are the basis for thinking about unintended consequences.
By this point, you should have enough thinking and inspiration that these flow out. All the work done to this point is the context for making these easy to write.
What exactly are unintended consequences? They are not outputs or use cases. They describe what we anticipate to be a negative knock-on effect. What could change in a harmful way because of your product? Again you want to make a connection to societal impact. You should encourage people to do this even if it seems far-fetched. It’s better to be far-fetched and than to assume there is no connection.
Bike example: Bikers presented higher rates of asthma and other respiratory problems as a result of being directly exposed to pollutants when travelling behind vehicles.
5. Risk mitigation
The final step may or may not be appropriate for a workshop setting. It is to prioritise and decide mitigation actions. Usually prioritising and mitigating is a strategic activity, and so those making those decisions should own this. Doing this within the workshop is a nice way of tying things up and it can be satisfying to see how all these thinking will make a difference. But it is something that the strategy people in the team will need time to think about. So you could go about this in two ways:
- Top-down: You may end the workshop on unintended consequences then work individually with the strategy people/PM to identify the actions, and then feed this back to the team for feedback.
- Bottom-up: Or if you think the team would benefit from something more bottom-up, you can ask for workshop attendees to brainstorm what to do to address the unintended consequences.
If you need a framework for prioritising I have used the severity versus impact plot, which is just one of many prioritisation tools already out there.
Another the framework for prioritisation, which DotEveryone uses, is ‘Act, Monitor, Influence’. I think these are great categories. You might find that some intended consequences will fit into more than one of these categories.
What you want to end up with is a list of feasible actions, and people know who is taking those actions forward.
Timings of analysis and workshops
Facilitating this process involves figuring out who to involve, how many workshops you want to do, and how much work you need to do in-between workshops.
The one lesson I have learned is that the workshop always takes longer than expected. And it is a fine line balancing being efficient with everyone’s time and keeping the conversation on track versus limiting natural and exploratory conversations from happening.
The minimum amount of time you need in workshop time is two hours. A healthier amount of time would be:
- One hour for intended consequences and use cases.
- One hour for filling out the use case quadrant.
- One hour for writing unintended consequences and actions.
I have sometimes needed to do offline work between workshops with product manager or other leads in order to clarify and strengthen the analysis.
Figure out what best suits your situation
You will have noticed by now that I have been giving a variety of options for facilitating this process. Prescribing an exact formula isn’t helpful. How you go about this analysis depends on where your team is at, how much consensus already exists, how defined and undefined things are. And be ready to change your approach as you go through the process and learn more.
