What is Algorithm Fairness?

An introduction to the field that aims at understanding and preventing unfairness in machine learning

Conor O'Sullivan
Towards Data Science
8 min readMar 5, 2021

--

(source: flaticon)

At first, the concept of an unfair machine learning model may seem like a contradiction. How can machines, with no concept of race, ethnicity, gender or religion, actively discriminate against certain groups? But algorithms do and, if left unchecked, they will continue to make decisions that perpetuate historical injustices. This is where the field of algorithm fairness comes in.

We explore the field of algorithm fairness and what it aims to achieve. To highlight the importance of this field, we discuss examples of unfair models and their consequences. We also briefly touch on the reasons for unfairness, how to measure it and how to prevent it. To end, we discuss the link between fairness and interpretability. Along the way, we have linked to more in-depth articles on these topics.

Before we move on, you can watch the video below for a summary of this article:

What is algorithm fairness?

In machine learning, the terms algorithm and model are used interchangeably. To be precise, algorithms are mathematical functions like linear regression, random forests or neural networks. Models are algorithms that have been trained on data. Once trained, a model is used to make predictions which can help automate decisions. These decisions can include anything from diagnosing a patient with cancer to accepting mortgage applications.

(source: flaticon)

No model is perfect, meaning they can make incorrect predictions. If these errors systematically disadvantage a group of people, we say the model is biased/unfair. For example, an unfair model could reject mortgage applications more often for women than men. Similarly, we could end up with a medical system that was less likely to detect skin cancer for black patients than white patients.

Algorithm fairness is the field of research aimed at understanding and correcting biases like these. It is at the intersection of machine learning and ethics. Specifically, the field includes:

  • Researching the causes of bias in data and algorithms
  • Defining and applying measurements of fairness
  • Developing data collection and modelling methodologies aimed at creating fair algorithms
  • Providing advice to governments/corporates on how to regulate machine learning

It is also important to understand that approaches to fairness are not only quantitative. This is because the reasons for unfairness go beyond data and algorithms. The research will also involve understanding and addressing the root cause of unfairness.

(source: flaticon)

Why is algorithm fairness important?

As mentioned, machine learning models are being used to make important decisions. The consequences of incorrect predictions could be devastating for an individual. If the incorrect predictions are systematic then entire groups could suffer. To understand what we mean by this, it will help to go over a few examples.

Apple recently launched a credit card — Apple Card. You can apply for the card online and you are automatically given a credit limit. As people started to use this product, it was found that women were being offered significantly lower credit limits than men. This was even when the women were of a similar financial position (and credit risk). For example, Apple co-founder, Steve Wozniak, said he was offered a credit limit 10 times higher than his wife.

(source: flaticon)

Another example is a system used by Amazon to help automate recruitment. Machine learning was used to rate the resumes of new candidates. To train the model, Amazon used information from historically successful candidates. The issue is that, due to the male dominance of the tech industry, most of these candidates were male. The result was a model that did not rate resumes in a gender-neutral way. It actually went as far as penalising the word “woman” (e.g. Captain of the woman’s soccer team).

These examples show that models can make predictions that discriminate based on gender. Women, who are equal to their male counterparts, are faced with significantly different results. The consequence, in this case, is a lower credit limit or the rejections of a job application. Both of these outcomes could have serious financial implications. In general, models like these will increase the economic inequality between men and women.

(source: flaticon)

Models can also discriminate based on race. COMPAS was an algorithm used by the American criminal justice system to predict if a defendant was likely to re-offend. An incorrect prediction (i.e. false positive) could result in the defendant being falsely imprisoned or having to face a longer prison sentence. It was found that the false positive rate was twice as likely for black offenders than white offenders. That is black offenders were twice as likely to be incorrectly labelled as potential re-offenders.

These examples show that we can find biased algorithms being used for different problems across many industries. The scale at which these algorithms make decisions is also a concern. A biased human is limited in the number of loans he could underwrite or people he could convict. An algorithm could be scaled and used to make all of the decisions. Ultimately, the consequences of a biased algorithm can be both negative and widespread.

The reasons for unfairness

Clearly, they are bad but how do we even end up with unfair algorithms? Algorithm fairness is actually a bit of a misleading term. Algorithms, by themselves, are not inherently biased. They are just mathematical functions. By training one of these algorithms on data, we obtain a machine learning model. It is the introduction of biased data that will lead to a biased model. That being said our choices around algorithms can still amplify those biases.

(source: flaticon)

Data can be biased for different reasons. The data we collect will reflect historical injustice which can be captured by a model (historical bias). Like with the Amazon hiring model, it could be due to a lack of minority representation (unbalanced datasets). It could also be due to model features that are associated with race/gender (proxy variables). We explore these reasons in more depth in the article below.

Analysing and measuring unfairness

Much of algorithm fairness research aims at developing methods to analyse and measure unfairness. This can involve analysing data for the potential reasons for unfairness mentioned above. It also involves measuring unfairness in model predictions.

Definitions of fairness

We can measure fairness in predictions by applying different definitions of fairness. Most of the definitions involve splitting the population into privileged (e.g. male) and unprivileged (e.g. female) groups. We then compare the groups using evaluation metrics. For example, under the equalised odds definition we require the true positive rates and false positive rates of the two groups to be equal. A model with significantly different rates is considered unfair. Other definitions include equal opportunity and disparate impact.

(source: author)

Exploratory fairness analysis

Assessing fairness does not start when you have your final model. It should also be a part of your exploratory analysis. In general, we do this to build some intuition around our dataset. So, when it comes to modelling, you have a good idea of what results to expect. Specifically, for fairness, you want to understand what aspects of your data may lead to an unfair model.

In the article below, we walk you through how to do such an exploratory fairness analysis. We also discuss definitions of fairness in more depth and show you how to apply them.

Correcting and preventing unfairness

If we discover that our model is unfair we would naturally want to correct it. Various quantitative approaches have been developed. We can divide them into pre-processing, in-processing and post-processing. This depends on what stage during the model development they are applied. For example, we can adjust the cost function of a regression model to consider fairness. This would be considered an in-processing method.

(source: author)

Quantitative approaches have their limitations. This is because fairness is a complicated issue. We need to see it as something that goes beyond our data and models. Ultimately, we also need non-quantitative approaches to fully address unfairness. These include addressing the root cause, awareness of the problem and team diversity.

We discuss more quantitative and non-quantitative approaches in the article below. These include examples of pre-processing, in-processing and post-processing methods. We also try to understand the limitations of these methods in more depth.

Interpretability and fairness

I love writing about algorithm fairness but most of my writing is about interpretable machine learning. Interpretability involves understanding how models make predictions. Fairness and interpretability are actually related. The main reason for this is that they are both about building trust in ML systems. We discuss this and the other 2 reasons in the article below.

I hope you found this article helpful! If you want to see more you can support me by becoming one of my referred members. You’ll get access to all the articles on medium and I’ll get part of your fee.

You can also find me on | Twitter | YouTube |

Image Sources

All images are my own or obtain from www.flaticon.com. In the case of the latter, I have a “Full license” as defined under their Premium Plan.

References

Birhane, A., (2021) Algorithmic injustice: a relational ethics approach. https://www.sciencedirect.com/science/article/pii/S2666389921000155

D. Pessach & E. Shmueli, Algorithmic Fairness (2020), https://arxiv.org/abs/2001.09784

Gal Yona, A Gentle Introduction to the Discussion on Algorithmic Fairness (2017), https://towardsdatascience.com/a-gentle-introduction-to-the-discussion-on-algorithmic-fairness-740bbb469b6

J, Vincent, Apple’s credit card is being investigated for discriminating against women (2019), https://www.theverge.com/2019/11/11/20958953/apple-credit-card-gender-discrimination-algorithms-black-box-investigation

S. Wachter-Boettcher, Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech (2017), https://www.goodreads.com/book/show/38212110-technically-wrong

The Guardian, Amazon ditched AI recruiting tool that favored men for technical jobs (2018), https://www.theguardian.com/technology/2018/oct/10/amazon-hiring-ai-gender-bias-recruiting-engine

Wikipedia, Algorithmic bias (2021), https://en.wikipedia.org/wiki/Algorithmic_bias

--

--

Towards Data Science
Towards Data Science

Published in Towards Data Science

Your home for data science and AI. The world’s leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

Conor O'Sullivan
Conor O'Sullivan

Written by Conor O'Sullivan

PhD Student | Writer | Houseplant Addict | Follow me for articles on IML, XAI, Algorithm Fairness and Remote Sensing

No responses yet