It is hard not to be amazed by everyday progress in AI/Machine Learning (Automation) news from technology giants, that is disrupting every industry globally. Stories ranging from deployment of self-driving cars, gaming (Watson-Chess & DeepMind-GO), speech recognition accuracy (Alexa, Google, Siri, …), recommendation systems (Netflix, Amazon, …), medical (Watson, …), fraud detection (Visa, Spam, …) etc. The core of Machine Learning is a sophisticated Mathematical Algorithm that predicts or optimizes a specific goal. The algorithms are everywhere and affect everything we do. In 2011, Marc Andreessen said that “software is eating the world”. The new phrase is, “Machine Learning algorithms are eating the world”. Some of the algorithms have a very positive impact and some have a very destructive impact, even if it was conceived with the best intentions.

“Weapons of Math Destruction”, an excellent book by Cathy O’Neal covers some of the algorithms that are destructive. Cathy is a mathematician and data scientist (algorithm junkie) and has experience in academia as well as in the private sector (hedge funds, risk analysis software, and advertising). She has an admirable background which provides very insightful commentary on what may be happening as an unintended consequence of some of these algorithms.

Throughout the book, she covers some of the algorithms that are used to assess teachers, approve loans, student acceptance into good schools, determining insurance rates, getting a job interview, jail term, policing neighborhoods, etc. She highlights how algorithms can feed on each other and affect the poor people.

“Poor people are more likely to have bad credit and live in high-crime neighborhoods, surrounded by other poor people. Once the dark universe of WMDs digests that data, it showers them with predatory ads for subprime loans or for-profit schools. It sends more police to arrest them, and when they’re convicted it sentences them to longer terms. This data feeds into other WMDs, which score the same people as high risks or easy targets and proceed to block them from jobs, while jacking up their rates for mortgages, car loans, and every kind of insurance imaginable. This drives their credit rating down further, creating nothing less than a death spiral of modeling. Being poor in a world of WMDs is getting more and more dangerous and expensive.”

“The math-powered applications powering the data economy were based on choices made by fallible human beings. Some of these choices were no doubt made with the best intentions. Nevertheless, many of these models encoded human prejudice, misunderstanding, and bias into the software systems that increasingly managed our lives. Like gods, these mathematical models were opaque, their workings invisible to all but the highest priests in their domain: mathematicians and computer scientists. Their verdicts, even when wrong or harmful, were beyond dispute or appeal. And they tended to punish the poor and the oppressed in our society, while making the rich richer.”

“Data is not going away. Nor are computers — much less mathematics. Predictive models are, increasingly, the tools we will be relying on to run our institutions, deploy our resources, and manage our lives. But as I’ve tried to show throughout this book, these models are constructed not just from data but from the choices we make about which data to pay attention to — and which to leave out. Those choices are not just about logistics, profits, and efficiency. They are fundamentally moral.”

What can we do? Firstly, we can raise awareness of the algorithms that impact the broader population and not become complacent to the decision-making data. After all, algorithms are mostly corporations’ opinions in code! Secondly, we can all push for independent audits of algorithms that are running under the hood and push for transparency. Finally, we can encourage a code of honor for algorithm developers to adhere to, to ensure that fairness is respected, implemented and adhered to. There is no simple solution. It is like a grass roots movement and the more people that participate, the better it becomes.

I hope you enjoy reading this book. Happy reading.

Hojjat Salemi