Unboxing AI Bias in Criminal Justice
More transparency is required in risk assessment tools to prevent discriminatory outcomes
This article is the first in a series for AI for Human Rights, a new publication in which Josefina Miró Quesada Gayoso and I will be exploring the challenges and opportunities that AI presents in advancing human rights. The views expressed are solely our own.
Our first story delves into the use and impact of artificial intelligence (AI) in one of society’s most critical functions: criminal justice. Broadly speaking, the purpose of criminal justice is to protect freedoms, punish offenders, and rehabilitate criminals through a system of law enforcement, prosecution, and prisons. Justice systems around the world have been integrating AI in their decision-making processes for over two decades, for a variety of tasks. Among these, AI is used in predictive policing to identify locations of repeated crime and try to predict where they would occur next. It is also utilized for surveillance, by monitoring, tracking, and storing information about individuals, such as through the facial recognition systems that are in place in airports or at immigration borders. In this article, we will focus on the role of AI in criminal risk assessment (we will explore other applications in future stories) and discuss why and how it can perpetuate existing biases. We will argue that a critical step towards mitigating these biases is ensuring a transparent use of AI.
Risk assessment in criminal justice refers to a set of data-driven tools, recently incorporating AI, that assess an individual’s risk of committing a crime. These tools, which produce a risk score as output, are employed at various stages of the criminal process, significantly impacting decisions made by courts, prisons, and by parole and probation officers. For instance, algorithms are used to predict the likelihood of misconduct by a defendant awaiting trial, influencing whether they should remain free during the legal process or be placed in pretrial detention. They are also utilized to determine the level of supervision needed for inmates upon release, based on the risk of them becoming recidivists — a term used to describe criminals who re-offend. Dozens of these risk assessment tools are currently in use and are integral parts of criminal justice systems worldwide, including in the UK and in 46 states in the US.
Now, how do risk assessment tools use AI to calculate risk scores? The answer, perhaps surprisingly, is that, aside from the companies that developed and own these algorithms, no one truly knows. What we know is that they are trained on historical crime data and that they use a variety of factors, such as socioeconomic status, family background, neighborhood crime, and employment status (among others), to reach a supposed prediction of an individual’s risk to commit a crime. Beyond that, they are what computer scientists refer to as a “black box” — a system that produces insights without revealing its internal workings. The lack of transparency is explained by the fact that many of these are proprietary algorithms, developed by for-profit companies that have no financial interest in disclosing their code. They are protected by trade secret law, meaning that the software and the data it uses are immune from third-party scrutiny. However, this raises questions about relying on for-profit companies to develop risk assessment tools, because it makes it difficult to fully understand their functioning, evaluate their fairness, and hold people accountable when things go wrong. And things have gone wrong.
Research has shown that the outcomes of risk assessment tools in criminal justice are biased, particularly against the race of the accused. But why does bias occur in AI models¹? Part of it could be attributed to the inherent bias that any algorithm has, generated by the social structures in which they are created. Trivially, if most programmers are white men, they could carry their socially ingrained stereotypes or prejudice. The other cause, which is likely the main one here, is bias related to data. AI models are trained on historical data, that is, on things that happened in the past. But if communities, such as Black African Americans, have historically been disproportionately targeted by law enforcement (for example because of heavier policing in predominantly black neighborhoods, or racist bias in the decision to make an arrest), there will be an overrepresentation of crime data recorded on them. AI models will learn from this skewed data, reaching the false conclusion that black people are more likely to commit crimes than, for example, white people, for whom fewer crimes are recorded. In other words, AI models reproduce and could even exacerbate the inequalities already ingrained in the criminal justice system. As a consequence, two individuals accused of the same crime may receive significantly different criminal risk scores based on factors that are beyond their control.
One technique to mitigate the bias that causes such discriminatory outcomes is to process the training data to achieve a more balanced representation of crimes recorded for different demographic groups. For instance, developers could undersample crime data involving black individuals and oversample data related to white individuals. Similarly, developers could adjust the algorithm to assign less weight to data points related to crimes perpetrated by black people compared to those committed by white people. While bias in AI, like in the real world, cannot be completely eliminated, these techniques, along with continuous monitoring and evaluation of model outputs, could help mitigate it and ensure that remedial actions are taken when it is identified. However, the challenge lies in the lack of transparency from companies developing these risk assessment tools. Without visibility into how they sample data and train their models, we have no way to know whether they are actively taking steps to mitigate bias.
One of the most (in)famous cases of bias in AI models is that of COMPAS, a risk assessment tool developed by Northpointe (now rebranded as Equivant) that has been widely used for criminal justice decisions in the US since 2012. COMPAS uses a set of factors, including some derived from responses to a lengthy questionnaire, to rank defendants on a scale of 1 to 10 into low, medium, or high risk of recidivism. While race is not explicitly one of those factors, many others that are considered, such as unemployment, poverty, or social marginalization, are closely correlated with race. For example, according to the US Borough of Labor Statistics, the unemployment rate for black people in the US is double that of white people, and when black people are employed they earn considerably less. Since unemployment and poverty predict re-offending, the algorithm will flag more black defendants as high risk even though it does not use race in the classification. Consequently, an AI model that uses these factors to calculate recidivism scores, without compensating for the inherent bias, risks discriminating against black people.
A 2016 study by ProPublica analyzing COMPAS found exactly this. ProPublica examined nearly 12,000 criminal defendants in Broward County, Florida, over a two-year period and discovered that, although COMPAS correctly predicted an offender’s recidivism 61% of the time, errors were more frequent for one racial group than for another. Black defendants were often predicted to be at a higher risk of recidivism than they actually were, while white defendants were often predicted to be less risky than they were.
The table illustrates that black defendants who ultimately did not re-offend over the two-year period were misclassified as higher risk 45% of the time, compared to 24% for their white counterparts. Conversely, white defendants who re-offended within the next two years were mistakenly labeled as low risk 48% of the time, against 28% for black re-offenders. If we were to evaluate the AI model in computer science terms, we would say that it generated more false positives for black people and more false negatives for white people. ProPublica argues that due to these discrepancies, the algorithm cannot be considered “fair”.
This leads us to one of the most intriguing topics at the heart of many AI ethics debates: algorithm fairness. We will have the opportunity to delve deeper into this subject in future articles, but for now, let’s analyze it in the specific context of COMPAS. Errors in the COMPAS algorithm’s predictions disproportionately affected black defendants. Despite 45% of them not committing another crime, the algorithm’s wrong prediction likely caused those black defendants to face harsher treatment from the criminal justice system. This occurs because, when predicting future crime, the model reinforces the bias already present in the criminal data, that is, black individuals are arrested more often than whites who commit the same offense. In our opinion, it is hard to argue that such an algorithm is fair unless there is clear evidence that steps were taken by Northpointe to counteract this bias. Without access to the code, it is an impossible task.
Eight years have passed since the publication of the ProPublica report, yet, based on our research, little has changed. COMPAS may still be biased, but we cannot tell. Black box AI tools continue to be used to make decisions, with the criminal justice system being just one of many examples. The solution is not to eliminate AI-powered risk assessment tools, let’s be clear. AI will continue to play an increasingly prominent role in the criminal justice system, as it dramatically improves efficiency and reveals patterns that humans wouldn’t otherwise be able to see. Furthermore, the problem of bias is not exclusive to machines; it equally affects human decision-making. The solution, that you will hear us repeat over and over again in this publication, is for humans and machines to work together, because they complement each other. However, this collaboration, like any other, requires transparency and trust. We must demand full transparency on AI models. We must be able to understand how they have been designed and what concrete steps have been taken to account for existing bias in the data, so that we can trust their outcomes. This is imperative in a field like criminal justice, where individuals’ freedoms are severely affected. Without transparency that enables us to mitigate bias, we will continue to see discriminatory outcomes, ultimately undermining the legitimacy of a criminal justice system founded on the principle of equality.
¹For simplicity, in this article, the terms ‘algorithm’ and ‘model’ are used interchangeably. However, in artificial intelligence, an algorithm is a procedure that is run on data to create a model. A model represents what was learned by the algorithm.
If you liked this story, follow our AI for Human Rights publication and don’t forget to share it with your friends and colleagues. Stay curious and see you soon!
