How We Can Fix This
Where should we start repairing these systems and the culture that produces them?
As scholar Virginia Eubanks, author of the book Automating Inequality, captures it, “If there is to be an alternative, we must build it purposefully, brick by brick and byte by byte.”
Some researchers and technologists are understandably calling for more holistic thinking about the kinds of systems and tools that are being built. The academic Kate Crawford, who specializes in the social impact of data systems, asks audiences, “What kind of world do we want to live in?” Princeton computer scientist Arvind Narayanan argues, “It’s not enough to ask if code executes correctly. We also need to ask if it makes society better or worse.”
Neither distributional nor representative forms of harm can survive without a cultural backdrop that enables them.
The sentiment is a good one — we should certainly work to make the world better for lots of people and not just a few. But this kind of call to action risks hollowness if it doesn’t focus on those people being hurt by discriminatory systems. As author Mandy Henk noted, our subjects aren’t some amorphous “they.” Instead, our discussions need to be grounded in the political and cultural contexts that make these people vulnerable or marginalized in the first place.
Others have suggested that both the collection of data and the design of data-driven systems must become more transparent, which will help ensure that everything is fair and representative. There are certainly cases where datasheets for AI or more robust peer review might be a good idea, but imagining ethical paths forward for research, big data, and algorithms means going beyond technocratic solutions. We have to work much harder to exercise patience, empathy, and humility in how we conceive of the lives and experiences of our data subjects.
To begin, engineers and data scientists are far more likely to create balanced and representational products if they themselves are from a diverse range of backgrounds. “If we don’t have diversity in our set of researchers, we are not going to address problems that are faced by the majority of people in the world,” said Microsoft research scientist Timnit Gebru. “When problems don’t affect us, we don’t think they’re that important, and we might not even know what these problems are because we’re not interacting with the people who are experiencing them.”
Engaging with humanities and social science projects can also help. To that end, it is welcome news that the Social Science Research Council recently announced that it’s partnering with Facebook to give independent researchers access to the company’s data. It’s a potentially huge step toward gaining a better understanding of how data-intensive social platforms like Facebook shape our world — and the lives of people in it.
Yet engaging other kinds of research isn’t enough. Social scientists have had their own share of high-profile controversies, from failing to secure the privacy of research subjects to experimenting with people’s emotions without their consent. But many others have succeeded in meaningfully focusing on the lives and experiences of their subjects, from indigenous and border communities’ use of fitness trackers and the impact of data collection on marginalized groups in major U.S. cities to online privacy and intimate partner abuse. Our engagement, then, must be well-considered and put the needs and vulnerabilities of specific groups first.
It is important to ask who might be damaged by a certain set of assumptions and a certain algorithmic solution. We should also ask: Who does this protect? Who cannot be a target of this set of data? I am shielding someone — what does that say about my system?
Most important, engineers and data scientists need to listen to and engage with affected communities.
They should understand the struggles and histories of vulnerable communities and be ready to challenge their own assumptions.
They should offer support and resources — not opinions.
They should support legislation, causes, and organizations that improve lives without making them increasingly dependent on data-intensive systems of tracking and control.
They should not draw on the lives and experiences of their subjects without contributing something in return.
Ultimately, we need empathy and thoughtfulness in the design of algorithms and data science if we are to change the damaging cultural narratives that reinforce injustice and inequality for vulnerable people.
Because it’s never “just” an algorithm. And there’s no such thing as “just” an engineer.