How To Run a Consequence Scanning Workshop
“By default, digital technologies are designed in ways that reproduce existing forms of structural inequality. Only through conscious and coordinated intervention can we bend the arc of digital technology development towards justice.”
— Sasha Costanza-Chock, Design Justice in Action
The responsibility to “bend the arc” toward justice has rarely felt as pressing as it does today. COVID-19 has thrown society for a loop. It has reshaped our relationships to place and time, to things we buy from the store, and to people we walk past on the street. Renewed awareness about persistent racial injustice has led to protests worldwide, and drawn our attention, once again, to the pressing work we need to do to combat structural inequality and make our systems more just. All of the cultural upheaval characterizing this moment in history has underscored the accountability we all must take to protect the people in our lives and our communities.
In the past, design has generally been accountable to the individual. What if we thought bigger, and considered societal implications from the get-go? We can harness design to help identify and mitigate ethical risk, and drive opportunities for positive impact.
Bringing ethics into the design process
“Once a product has been fully conceptualized and built, much less shipped, it’s far too late to ask questions like ‘should this exist?’ or ‘could this be used in a harmful way?’. You accrue ethical debt that is extremely difficult, if not impossible to pay back.” — Kathy Baxter, Architect, Ethical AI, Salesforce
Ethical considerations are, by their very nature, challenging to grapple with. But examining the ambiguous space between right and wrong, good and bad, can be core to design’s function in organizations. To reframe objectives and drive ethical decision-making, designers are positioned to interrogate products, features, and services before they are put out into the world. If we don’t pause and consider the ramifications of our choices, we’re less able to prevent misuse and protect people and communities from harm.
Consequence Scanning is a framework that purposefully inserts friction into the product development process, with the goal of mitigating negative or unintended consequences and identifying opportunities for impact. It offers a framework to think through a product’s positive impact, and any negative impacts that might be prevented.
Consequence Scanning was developed by UK-based think-tank Doteveryone, which was founded five years ago to advocate for responsible technology and innovation. Since then, Doteveryone has contributed groundbreaking research, policy advice, solutions, and thought-leadership, all of which have pushed the conversation about technology’s impact on society forward.
On May 28, 2020, Doteveryone announced that they had chosen to end their work, and would be passing the baton to a set of nonprofit organizations to continue their mission. As part of that transition, they also handed technology creators the resources they had developed and collected. Salesforce continues to implement frameworks and processes created by Doteveryone in the work led by our Office of Ethical and Humane Use of Technology. We are deeply grateful to Doteveryone for their leadership and creativity, and the ways they’ve pushed the space forward.
How to run a Consequence Scanning workshop
“Consequence Scanning is a lightweight, agile methodology that gets teams accustomed to flexing their ethical foresight and building moral muscle memory.” — Yoav Schlesinger, Director, Ethical AI Practice, Salesforce
Consequence Scanning takes the form of a workshop. At Salesforce, we have begun to implement Doteveryone’s framework as a means to build up “moral muscle memory” in our product development process. We do this to make sure we are living our values, and to develop features that identify and mitigate ethical risk for our customers and end users. We’ve created this “cheat-sheet” that distills the Consequence Scanning framework, including insights and slight tweaks we’ve learned through applying it.
Setting it Up:
- Use a narrow scope: Instead of boiling the ocean, choose a particular feature or aspect of a product to interrogate. Narrowing in on specific areas of concern allows a deeper dive and more meaningful conclusions.
- Engage diverse perspectives: We all have biases — obstructions to our perspectives, borne out of our particular experiences — that may limit our ability to spot ethical risk and opportunity. Bring together a cross-functional group with varied experiences to collaborate.
- Assign some homework: Time is always a limiting factor, especially in fast-paced business settings. Ask people to come to the workshop with a list of potential consequences they have considered individually to keep the workshop as efficient as possible. We typically schedule 90 minutes.
Running the Workshop:
- Brainstorm about all the intended and unintended consequences of your new tool or feature. Write them down. Share them with your group.
- Map potential consequences on a 2x2 axis.
- Prioritize, prioritize, prioritize. What are the positive consequences you want to focus on? What are the negative consequences you want to mitigate? What’s doable? What’s a must have? What must be done now and what can be saved for the future?
Making it Stick:
- Make a plan. Assign individuals to lead the charge for each consequence. Make sure these outputs get integrated into your current roadmap or other existing processes.
“This was a great exercise for me to learn more and think more critically…For the particular use case that we discussed, it illuminated that the most obvious benefits were for our customers and ourselves and doing the exercise helped us be more thoughtful about the end user too. Knowing this framework will help me be more critical of what we are building.” — Workshop participant
The purpose of these workshops is to get teams thinking about their impact. Ethical awareness is a habit like any other. We have to be proactive about implementing the practice until it becomes second nature to question our beliefs and assumptions about the impact of our choices. When we all slow down, even if it’s just for 90 minutes, to consider, “What if?” we are able to take part in the important work of reshaping our systems and societies so that more people are protected from harm.
The Path Forward
“Consequence Scanning adds meaningful and positive friction into the design process to ensure everyone slows down just long enough to ask the really tough questions about a product and its potential uses, including questions about who else we need to consult, whether users, non-users, or advocates.” — Emily Witt, Senior User Researcher, Research & Insights, Salesforce
When we bring risks out in the open, it creates space to think creatively about how to mitigate them. We all get to be innovators when we design technologies that enable humans to improve the the world and discourage users from causing harm. Consequence Scanning is just one way Salesforce is working to bring ethics into our product development lifecycle and identifying opportunities for positive impact.
When it comes down to it, this stuff is really hard. We cannot claim to have all the answers here, and are looking to experts across industries (such as those who have contributed to Doteveryone’s work over the years) as well as impacted communities to listen, learn, and grow as an organization.
Looking ahead, we are continuing to invest in partnerships and coalitions to increase our awareness and accountability for the consequences of our choices. We commit to sharing resources we’ve found useful, and insights we believe can help people tackle these challenges in their own fields and organizations.
Resources:
- To learn more about our approach and commitment to the preservation of human rights, check out the website of our Office of Ethical and Humane Use of Technology.
- Check out the Privacy and Ethical Use Principles we put together to guide our COVID-19 Response.
- Take the Responsible Creation of Artificial Intelligence Module on Trailhead to learn how to remove bias from your company’s A.I. systems.
Learn more about Salesforce’s design philosophy and culture at www.salesforce.com/design.
Follow us at @SalesforceUX.
Want to work with us? Contact uxcareers@salesforce.com
Check out the Salesforce Lightning Design System
