On criminal justice and “Civilian Casualties”
\reading Weapons of Math Destruction\
In her book Weapons of Math Destruction, Cathy O’Neil offers an engaging and informative account of the many ways in which algorithms are — largely unchecked — redefining basic social concepts like equality, fairness, and justice. Aside from demonstrating that these “WMDs” are already being deployed to rank teachers, sort job applicants, rate credit, and filter our news, O’Neil presents the reader with three criteria to determine if the latest application of “big data” amounts to a “weapon of math destruction”: is it opaque, does it operate at a large scale, and does it cause real-world harm? If the answers to all three questions are yes, you’re dealing with a WMD.
This elegant characterization is woven throughout the book and is a real strength, as it enables the synthesis of many different and complicated examples. As someone who studies the US criminal justice system, I was particularly drawn to the fifth chapter of Weapons of Math Destruction, titled “Civilian Casualties.” Here, O’Neil illustrates the way that WMDs and their relatives are shaping policing, the courts, and incarceration. The four main examples she provides — PredPol, stop-and-frisk, recidivism scores, and Chicago’s “heat list” — touch on issues of poverty, fairness, racism, and surveillance.
“Civilian Casualties” begins with a discussion of PredPol, a Santa Cruz based company whose hotspot-policing software, despite good intentions, provides what O’Neil describes as a “do-it-yourself WMD” for police departments around the country. She pursues questions here and throughout the chapter about how crime is defined, how it’s detected, and how crime-control measures are enforced. O’Neil makes the case that by including quality-of-life offenses in the data used to train PredPol’s software (i.e., in selecting an expansive definition of crime), police go to poor areas more and detect more such offenses. By making more arrests for those quality-of-life offenses, an undue burden is placed on residents of poor areas. Meanwhile, white-collar crime committed by residents of wealthy areas is ignored; poverty is, in effect, criminalized.
This argument, however, is incomplete. Crime is notoriously difficult to measure, and police resource allocation decisions — at all levels — are often made on the basis of reported crime rather than crime observed by officers. A discussion of where and why people call 911, and associated concerns such as underreporting and community distrust of police, would have provided a deeper view into the issues of crime definition and detection.
The next example O’Neil presents is New York’s stop-and-frisk policy, where officers briefly detain an individual given reasonable suspicion that crime is afoot, and conduct a pat-down if they suspect the individual is armed and dangerous. As O’Neil points out, stop-and-frisk isn’t technically a WMD, since it’s not implemented as an algorithm, but it’s similar to one.
After tracing the history of stop-and-frisk, she wonders whether “we as a society are willing to sacrifice a bit of efficiency in the interest of fairness.” In other words, even if WMD’s and their ilk do reduce crime, are the social costs they impose worth the benefits?
She makes a compelling argument that the answer is often no. What isn’t mentioned, though, is that in the case of stop-and-frisk, fairness and efficiency are actually aligned with each other, because most stops in New York City uncovered no criminal activity (90% of stops resulted in no further police action such as a citation or arrest). In this situation, one could use an algorithm to make better stops, simultaneously recovering illegal weapons, eliminating unnecessary intrusions on citizens, and reducing racial disparities. In a recent paper, my collaborators and I showed that using a transparent model — a simple, three-item checklist — to inform stop decisions, police could make tens of thousands fewer stops while retaining many of the crime-control benefits of stop-and-frisk. This type of model might serve as a template for the good uses of algorithms and big data, the opposite of a weapon of math destruction.
There are a few minor mischaracterizations and omissions in this chapter of Weapons of Math Destruction that I would have liked O’Neil to address. CompStat is not, as she suggests, a program like PredPol’s. This is a common misconception; CompStat is a set of organizational and management practices, some of which use data and software. In the section on stop-and-frisk, the book implies that a frisk always accompanies a stop, which is not the case; in New York, only about 60% of stops included a frisk. Moreover, the notion of “probable cause” is conflated with “reasonable suspicion,” which are two distinct legal standards. In the section on recidivism, O’Neil asks of prisoners,
“is it possible that their time in prison has an effect on their behavior once they step out? […] prison systems, which are awash in data, do not carry out this highly important research.”
Although prison systems may not conduct this research, there have been numerous academic studies that generally indicate a criminogenic effect of harsh incarceration conditions. Still, “Civilian Casualties” is a thought-provoking exploration of modern policing, courts, and incarceration. By highlighting the scale and opacity of WMDs in this context, as well as their vast potential for harm, O’Neil has written a valuable primer for anyone interested in understanding and fixing our broken criminal justice system.
Ravi Shroff is a researcher at NYU’s Center for Urban Science and Progress and a 2016–2017 Fellow at Data & Society. His work is broadly related to computational social science, and in particular the application of machine learning techniques to a variety of urban issues.
Points/WMD: Together and individually, the Data & Society community has been reading Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, which concerns many areas of our work and research — and we’re posting our responses to Cathy’s book, mini-symposium-style. More here:
- Models in Practice by Angèle Christin
- Safety Checklists for Sociotechnical Design by Mark Ackerman
- Shining a light on the darkness by Mark Van Hollebeke
- Sausage, politics, and data predictions? by Anne L. Washington
- Ethics review for pernicious feedback loops by Jacob Metcalf
Cathy also recently spoke at Data & Society. Video here. — Ed.
