Shining a light on the darkness
\reading Weapons of Math Destruction\
Cathy O’Neil’s stated goal in Weapons of Math Destruction is to help us see “the dark side of Big Data” (13). She pursues that objective with a readable and engaging writing style and with stories of real people whose lives have been damaged (sometimes irreparably) by her so-called WMDs.
Let’s be clear, we can no longer reject the existence of WMDs. O’Neil has found them, and they were not in Iraq.
The weaponized mathematics she describes results from “opaque, unquestioned, and unaccountable” algorithms currently deployed to “optimize” everything from teacher evaluations to criminal sentencing and hiring practices. Her version of WMDs serve as a compelling and accessible interpretive lens, and the damage she reveals is palpably felt by the reader.
She shows us how a “value-add” teacher evaluation algorithm is being used in school districts all over the U.S. to summarily dismiss otherwise highly-regarded teachers. And yet, there’s no explanation of how the model works. It’s a black box operating with zero accountability. If an evaluation is truly meant to enhance teacher performance and improve the quality of education shouldn’t it be open to evaluation itself?
If that example leads you to scratch your head in frustration, her next will make you shake your fist in rage. As a quant at DE Shaw, O’Neil witnessed firsthand how mathematical models could mask the truly toxic nature of mortgage-backed securities that lead to the 2008 financial meltdown. Her explanation of this math-made global financial crisis is succinct and cogent.
The ability of “crack mathematicians” to (allegedly) mitigate risk was being deliberately used to perpetuate fraud on consumers who would be left holding the bag as banks managed to “unload the securities before they explode” (41). That was the intent of the bankers. But even they failed to realize the destructive force of the WMDs they deployed.
This book should be required reading for all data scientists and for any organizational decision-maker convinced that a mathematical model can replace human judgment. Not because we should reject data science and algorithmically-driven analytics, but because of how important carefully designed and principled data science is.
O’Neil clearly demonstrates the damage that our age of unquestioned techno-optimism, combined with slipshod mathematical modeling and cavalier automation can produce. What we need now is good data science coupled with good social science. And the realization that you can’t have good data science without humility and principled design practices. What’s more, the IT industry needs to take on enhanced accountability for the powerful tools it is creating.
What controls (and yes, regulations!) are required to make ethical data use the norm, rather than the exception?
O’Neil begins to articulate these very points in her conclusion. My qualm with this book it is that she takes too long to start saying things like, “As I survey the data economy, I see loads of emerging mathematical models that might be used for good and an equal number that have the potential to be great — if they are not abused” (216). Prior to seeing this statement, a reader unfamiliar with data science could be forgiven for coming to believe that all algorithmically-driven processes are evil.
Why not clearly delineate what good data science looks like very early on, and explain how WMDs can be disarmed (sooner than she does) and how mathematical models can be used for good? I want O’Neil’s book not simply to frighten us, but to show us how we can course correct before real damage is done. But this may be too much to ask of one book and one person. As she points out, “our society is struggling with a new industrial revolution” (204). Although Upton Sinclair helped expose the problems of the previous industrial revolution, he did not resolve them on his own. Weapons of Math Destruction is The Jungle of our age.
As a privacy professional at a major technology multinational, I am much closer to being a Big Data evangelists than O’Neil is, but like many other engaged privacy professionals, I am equally concerned about the challenges that algorithmically-driven analytics present as the promise they hold. It is my sincere hope that O’Neil’s book will help the general public, policy makers, and IT professionals to see the problems and realize the full scale of the risks she explains. Data analytics inflicts harm on the most vulnerable and exacerbates existing injustice when we allow “ill-conceived mathematical models [to] micromanage the economy.” That said, we should not give up all hope. O’Neil’s book should be read as a call to action.
Adopting clear data ethics practices, and creating legal enforcements of them, would preserve respect for individual agency and community self-determination — the loss of which, O’Neil bemoans — and could contribute to the improvement of societal equality and diminish power asymmetry between large institutions and individuals.
O’Neil’s analysis doesn’t just apply to mathematical models; it applies to societal models. Most of the WMDs that Cathy O’Neil describes are inextricably linked to unjust social structures.
We all, data scientists included, need to act with some humility and reflect on the nature of our social ills. As O’Neil writes, “Sometimes the job of a data scientists is to know when you don’t know enough” (216). Those familiar with Greek moral philosophy know that this type of Socratic wisdom can be very fruitful.
It’s not just the dark side of Big Data she shows us, but shady business practices and unjust social regimes. We will never disarm the WMDs without addressing the social injustice they mask and perpetuate. O’Neil deserves credit for shining a bright light on this fact.
Mark Van Hollebeke is a privacy professional at Microsoft. At Data & Society he is working to find practical ways to interject ethical reflection into the data-analytics design process. A former philosophy professor specializing in ethics, pragmatism, and social and political philosophy, his goal is to broaden existing IT industry privacy practices to include moral inquiry about the nature of data use.
Points/WMD: Together and individually, the Data & Society community has been reading Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, which concerns many areas of our work and research — and we’re posting our responses to Cathy’s book, mini-symposium-style. More here:
- Models in Practice by Angèle Christin
- On criminal justice and “Civilian Casualties” by Ravi Shroff
- Safety Checklists for Sociotechnical Design by Mark Ackerman
- Sausage, politics, and data predictions? by Anne L. Washington
- Ethics review for pernicious feedback loops by Jacob Metcalf
Cathy also recently spoke at Data & Society. Video here. — Ed.
