Weapons of Math Destruction (O’Neil, 2016)

How big data increases inequality and threatens democracy

A review by Jimena Villacorta

You can read the Spanish version here.

Mathematician, data scientist and author Cathy O’Neil has been one of the loudest advocates in favor of restricting the influence of algorithms over our lives, opposing the assumption that algorithms are “impersonal” and therefore fair and unbiased. Her book, Weapons of Math Destruction, explores the bias present in data analysis, the unintended consequences of big data and the ethics surrounding it — aspects we tend to ignore when thinking about technology.

Algorithms are increasingly affecting people’s lives to the point where decisions involving where to attend college, job applications, health insurance, getting a loan or voting are no longer controlled solely by us humans, but also by mathematical models. Some believe this use of algorithms implies greater fairness, as they are said to eliminate prejudice by judging everyone by the same rules.

However, O’Neil characterizes these models, which she calls weapons of math destruction (WMD), as both widespread and inconspicuous, as nobody seems to really notice or care that they make decisions on behalf of many people. She also calls them mysterious, because it’s not clear how they are made and people who are targeted by these models don’t know how they work. Finally, O’Neil labels these models as destructive, for despite being set up to solve problems, they often make them worse by being unfair and biased towards certain people.

Weapons of math destruction are algorithms used in all types of companies as a form of social control which targets the most vulnerable by codifying racism or prejudice
— Cathy O’Neil

Among the examples of WMD O’Neil discusses, there are three in particular which showcase the author’s message throughout the book, being how these algorithms have unexpected consequences in people’s lives because of their unpredictability and people’s inability to understand them completely.

Education

She first speaks about a widespread education reform algorithm meant to hold teachers accountable for good teaching that got teachers in Washington DC and Chicago fired. O’Neil criticizes the algorithm for having little accountability and yet scores teachers’ accountability in the system, sharing that it was even responsible for firing a woman she knows based on value added model scores that nobody can really understand or improve because it is forbidden for anyone in the education system to see inside of it.

Justice

O’Neil also discusses the justice system, in which there are four tiers. First, data comes from policing events which as we know is uneven, like the stop-and-frisk in New York, as she explains that these techniques take into consideration external factors such as race. Next comes predictive policing and evidence-based sentencing: when someone is found guilty, the judges ask for a recidivism risk score to decide how long that person should go to jail.

O’Neil criticizes these models as being embedded opinions which promote prejudice. O’Neil writes, “A 2013 study by the New York Civil Liberties Union found that while black and Latino males between the ages of fourteen and twenty-four made up only 4.7 percent of the city’s population, they accounted for 40.6 percent of the stop-and-frisk checks by police. More than 90 percent of those stopped were innocent”. O’Neil accuses these models of punishing the most vulnerable of society and criminalizing poverty.

Politics

The next example O’Neil discusses are the microtargeting politics aimed at understanding voters. Campaigns use microtargeting to identify what voters want, to learn what their profiles are like, testing people’s behavior through different messages and advertisements in social media, and use this to get to vote for a certain candidate. “The growing science of microtargeting, with its profiles and predictions, fits all too neatly into our dark collection of WMDs”, O’Neil writes, “…it is vast, opaque, and unaccountable. It provides cover to politicians, encouraging them to be many things to many people”.

By providing a greater understanding of how we are often blindsided by these mathematical models, Weapons of Math Destruction encourages us to change our habits when it comes to technology. While O’Neil wants more responsible models and greater regulation of its use by politicians, she makes it clear that it is up to each one of us to take responsibility for how we allow them to influence our lives.

Weapons of Math Destruction is a very technical book filled with a wide range of statistics; yet it is also entertaining and very engaging until the end, where O’Neil proposes ideas on how to fight these WMD and use big data to improve people’s lives and promote equality and justice, rather than threaten them. O’Neil writes, “Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have to explicitly embed better values into our algorithms, create Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.”

Jimena Villacorta studies International Relations at the University of Navarra| LinkedIn