Sausage, politics, and data predictions?
\reading Weapons of Math Destruction\
If no one should see how sausage and politics are made, should anyone see how data predictions are made? Cathy O’Neil says yes in the new book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. O’Neil explains how practices taken from business analytics can be disastrous when applied without additional rigor to public policy.
Businesses generate marginal profits by predicting buying patterns based on millions of transactions. What happens when that mathematical firepower calculates classroom performance using a statistical sample size of 30 students? O’Neil uses her expertise as a data scientist to question predictions that cause teachers to lose their jobs or significantly impact other aspects of society.
The book challenges the problem definitions underlying public policy analytics. A problem definition, according to political scientist Deborah Stone, is a package of perspectives that express a causal story about who is to blame for a social condition. Smoking, for instance, became a public health issue when politicians defined the problem around second-hand smoke instead of individual choice.
O’Neil suggests that narrowly defining student achievement through teacher performance does not meet the goals articulated in the associated education policy. In this and in other cases, she offers alternative ways to model problems.
Opposing problem definitions traditionally frame policy solutions. This book opens the possibility for a new type of policy analysis where opposing policy definitions are framed by competing data models and metrics.
The book’s engaging stories focus solely on data predictions gone wrong and question whether we can learn from these failures. Auditing is one remedy O’Neil offers. Although auditing can be seen as a punitive measure, it has a positive side as well. Superior data scientists would be able to differentiate their models from mediocre ones.
Models, O’Neil reminds us, are dynamic instruments that require tinkering to maintain excellence.
Complex models with high stakes require rigorous periodic taste tests. Unfortunately most organizations using big data analytics have no mechanism for feedback because the models are used in secrecy.
Producing predictions, like making sausage, is currently an obscure practice. If botulism spreads, someone should be able to identify the supply chain that produced it. Since math is the factory that produces the sausage that is data science, some form of reasoning should be leveraged to communicate the logic behind predictions.
Weapons of Math Destruction provides many cautionary tales about predictions used in public policy. Organizations that greedily consume data analytics without asking questions may find the expected flavors delivered with the aroma of efficiency only temporarily satisfying.
Anne L. Washington is a computer scientist and a librarian who specializes in public sector technology management and informatics. She is an Assistant Professor at George Mason University and a 2016–2017 Fellow at Data & Society.
Points/WMD: Together and individually, the Data & Society community has been reading Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, which concerns many areas of our work and research — and we’re posting our responses to Cathy’s book, mini-symposium-style. More here:
- Models in Practice by Angèle Christin
- On criminal justice and “Civilian Casualties” by Ravi Shroff
- Safety Checklists for Sociotechnical Design by Mark Ackerman
- Shining a light on the darkness by Mark Van Hollebeke
- Ethics review for pernicious feedback loops by Jacob Metcalf
Cathy also recently spoke at Data & Society. Video here. — Ed.
