Ethics review for pernicious feedback loops
\reading Weapons of Math Destruction\
Cathy O’Neil’s Weapons of Math Destruction offers us a series of parables about the often-unintended consequences of leveraging algorithms for social, political and economic goals. This book takes an important insight from science and technology studies and applies it to the data analytics revolution: although it is true that all technologies can be used for good and ill, it is also true that technologies always operate within value-laden sociotechnical systems. Just like nuclear power or biotechnologies, algorithms embody the values of the systems that enable their various uses. We’re mistaken if we take a naive view that algorithms are truly neutral forces, let alone if we buy into the world-saving hype out of Silicon Valley.
What this book adds to the conversation in science and technology ethics is highlighting a complicated and unique feature of algorithms. Unlike a nuclear power plant or biotechnology, algorithms applied to human social and economic behavior create feedback loops that reinforce their own justifications for use.
Algorithms not only model, they also create. For researchers grappling with the ethics of data analytics, these feedback loops are the most challenging to our familiar tools for science and technology ethics.
For example, in her discussion of predictive analytics in policing, O’Neil describes how the service PredPol splits crimes into two different classes with potentially dramatic effects: violent and property crimes on the one hand, and nuisance crimes on the other. The patrol officers using PredPol on their tablets are shown a highlighted small geographical areas that the algorithm predicts will likely be a site for a crime, and so officers spend more time in that area. PredPol’s founders market their product in part by noting that no demographic data is used to make any predictions. The historical datasets provided by the local police contain only the what, where and when of past crimes, and never the who. This ostensibly shields their predictions — adapted from an earthquake aftershock prediction algorithm — from the sorts of personal bias that officers may have.
And yet this seemingly neutral dataset creates a self-reinforcing feedback loop that retroactively justifies attention paid to nuisance crimes, such as aggressive panhandling, loitering or public intoxication. Nuisance crimes are overwhelmingly cited in economically depressed and predominantly non-white areas. For example, there are not necessarily more intoxicated persons in poor urban neighborhoods than in the suburbs, but there are a lot more intoxicated persons who lack a private place to imbibe. As O’Neil points out, public intoxication is the sort of quality of life crime that police are rarely called by a citizen to deal with, but rather is policed primarily when officers have the time and inclination to notice it happening. Nuisance crimes are also much easier to accurately predict than violent and property crimes because they are both more common and have a much stronger geographic clustering. Thus a system that quantifies and predicts when and where nuisance crimes will occur cannot help but be a proxy for who is committing the crime: if all the whos live in one place, then where is a proxy for who, whether that is intended or not.
Ultimately, as has been demonstrated by the court cases over NYPD’s stop and frisk program and the Ferguson traffic court’s outsized warrant load, even nuisance crime enforcement has the potential to have enormously consequential disparate impacts along racial lines. What O’Neil does so well is to show us how even if we set out to use neutral datasets, algorithms applied inside an unjust environment create a “pernicious feedback loop … [that helps] to create the environment that justifies their assumptions. This destructive loop goes round and round, and in the process the model becomes more and more unfair” (pg. 29). In this way, a crime prediction algorithm is not only modeling or mathematically representing the world, it is also instantiating its modeled assumptions in the behavior of those who act on the predictions. When nuisance crimes are policed they generate more data, more data means more accurate predictions, which means nuisance crimes are policed more accurately, which means where is an ever stronger proxy for who, ad infinitum, until what is modeled (nuisance crimes happen here) becomes reality (these people attract disproportionate policing efforts). What is ultimately a value decision — we choose to police these crimes for those reasons — becomes naturalized through the black-box “objectivity” of the algorithm.
From a research ethics perspective, this feedback loop that moves algorithms from modeling to instantiating is a major difference between algorithms and more familiar technologies, and we are ill-prepared to deal with it. As I’ve argued elsewhere, there are a number of disjunctions between existing research ethics norms and regulations and the methods and consequence behind data analytics.
“Research ethics” is not just the study of the right thing to do in scientific research, it is also an institutionalized set of norms and practices that establishes our shared expectations about how science can or should be governed.
When it comes to research ethics in the U.S., the primary influencer in research ethics is the Common Rule, which requires federally funded human-subjects research to be reviewed by Institutional Review Boards (IRB) to ensure that certain kinds of risks are mitigated and that research projects meet certain standards when appropriate (such as informed consent). Due to historical quirks in how “human-subjects” and “research” are defined by those regulations, most of the activities we call “data science” fall outside of those regulations, and data science receives little in the way of prior ethics review. For example, the Common Rule and IRB precedents only attempt to measure and mitigate harm done to individuals directly by the research activities, not downstream and distributed consequences for populations created by the pernicious feedback loops described by O’Neil. Additionally, most of the influential algorithms in the wild today are only distantly the result of academic research that falls under the Common Rule in the first place.
That leaves data scientists, ethicists and developers in a challenging spot. Not only is most of data science formally outside of research ethics infrastructures, we also lack the informal norms and habits of mind that set expectations about how these technologies should be reviewed and their harms mitigated. If a group of data scientists were genuinely worried about an instance of pernicious feedback loops they would presently lack a clear model of how to handle it. What would it even look like for a predictive policing company or their potential clients in the justice system to audit for a potential pernicious feedback loops? Furthermore, what would it look like to shut down a feedback loop once it was identified? We are only beginning to articulate these peculiar ethical challenges of data analytics.
Mark Ackerman’s discussion of “safety checklists for sociotechnical design” has some very helpful starting points. Facebook made a fairly bold (if incomplete) step in corporate research ethics this year as they described their internal review practices for approving new research projects and product design. O’Neil offers only a few suggestions about how data scientists might mitigate the harms created by their products, many of which are more philosophical than practical. However, for my money, the most radical claim in WMD is the suggestion that data scientists may have a duty to destroy certain datasets in order to avoid pernicious feedback loops. Contrary to the spirit of “Big Data” that favors ever more data, it’s becoming clear that sometimes just outcomes needs to win out over accuracy of predictions in order to avoid pernicious feedback loops.
Jacob Metcalf, PhD, is a consultant and independent scholar specializing in data and technology ethics. His academic background is in applied ethics, particularly in science and technology. He is also a researcher at Data & Society.
Points/WMD: Together and individually, the Data & Society community has been reading Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, which concerns many areas of our work and research — and we’re posting our responses to Cathy’s book, mini-symposium-style. More here:
- Models in Practice by Angèle Christin
- On criminal justice and “Civilian Casualties” by Ravi Shroff
- Safety Checklists for Sociotechnical Design by Mark Ackerman
- Shining a light on the darkness by Mark Van Hollebeke
- Sausage, politics, and data predictions? by Anne L. Washington
Cathy also recently spoke at Data & Society. Video here. — Ed.
