AI Clarified: Is AI More Biased Than Humans or Less?

data4help
18 min readOct 27, 2020

Exploring bias in AI systems, and what we can do to prevent it.

About the AI Clarified series

For business and non-profit leaders trying to understand AI, it can be surprisingly difficult to find good information in the sweet spot between high-level overview and technical jargon. The AI Clarified series attempts to fill this void and answer some of the most commonly asked AI questions with practical, easy-to-follow explanations.

To Clarify: The question of bias in AI

Question: Is AI more biased than humans, or less? I’ve heard both and am not sure which side to believe.

Indeed it’s hard to know what to believe about bias in Artificial Intelligence (AI) systems when just reading articles online — there is plenty of support in both directions. With the growth of AI and the widespread adaption of AI models, there is a lot of noise on both sides, especially for high-stakes use cases like those affecting humans.

Let’s take hiring as an example. Hiring is an area that is high-stakes and clearly affects people directly. It is also an area we know is highly flawed — much research has been done about the human biases that cloud the hiring process. For example, one study found that the exact same resume with a “black sounding” name received drastically fewer callbacks than one with a “white sounding” name.

In response to these problems with traditional hiring processes, and because of the ease of implementing AI systems today, the call to replace flawed human recruiters with AI have grown louder. Harvard Business Review, for example, has advocated for the wide-spread adoption of AI systems in hiring processes, saying that they can remove the unconscious biases that plague human hirer. At the same time, companies who have tried to apply AI to their hiring processes have had to halt their experiments due to the AI systems also being biased themselves.

This leads us to our Question to Clarify: Which side should we trust more, humans or machines? If we want an unbiased decision, which side should we trust to make it for us?

The short answer is that AI systems can be either more or less biased than human decision-makers. The rest of this post constitutes the long answer — how biases appear in AI systems, how can spot them, and how we can prevent them from appearing in the first place.

Sticking with the example of AI in hiring, we first talk about how AI systems learn to make decisions. Then we apply this understanding of machine learning, the technology which underpins AI systems, to the specific problem of hiring and use an example dataset to show how the way we prepare the data for modeling can have a huge impact on the bias it exhibits at the end.

How does AI make decisions?

The decisions that AI systems make are entirely based on the data used to train them. Bias comes from training the AI systems with data containing biases. In this section, we will walk through how AI systems work and how bias can be “baked in” to them.

To start, we need to look at the vocabulary we are using and clarify the terms. “AI systems” are actually made up of machine learning algorithms, which are trained with input data. A trained algorithm is called a model. “Machine learning algorithms” sound complicated, but it is actually quite simple.

The goal of machine learning algorithms is to identify the relationship between inputs and an output, what they’re trying to predict. AI systems most often complete a certain task — predicting one output. In the example of hiring systems, what the AI system does is actually predicts whether or not a person would be hired, based on past hiring decisions.

To better understand how this works, let’s look at how one machine learning algorithm actually works, and how it arrives at predictions. The algorithm we will explore in more detail is called a Decision Tree.

All machine algorithms, including Decision Trees, learn from sets of input data called training data. This training data can be broken down into 2 parts: the target, what we’re trying to predict, and features, the data that helps the algorithm predict the target. The data below shows a simplified problem for machine learning: predicting whether or not a person played golf, based on input features like the weather outlook, temperature, and humidity.

Training data for example machine learning problem

All machine learning algorithms then work iteratively through this training data to identify and “record” the relationship between the input features and the target output. This is where the term learning in machine learning comes from: we do not explicitly tell the algorithm that every time the outlook is Overcast, the person plays golf. Rather, the algorithm learns that relationship on its own as it iteratively works through the data, updating its understanding as it goes.

This distinction — that the machine learns these relationships on its own and is not explicitly told them through explicit rules — is really what makes machine learning so powerful. Especially for problems with very many features, machine learning thus allows us to discover relationships between feature data and the target that would likely be too complex to program by hand, or that we would possibly not even be able to recognize on our own. Machine learning models are thus often more accurate than human-made models.

There are dozens of different machine learning algorithms that we could use to model the relationship between the features and target and get our predictions. These algorithms only differ in how exactly this relationship is “recorded”. We’ll now look at how the Decision Trees algorithm identifies, and ultimately saves and applies, the relationships between features and the target. We’ll use a simple use case and simplified form of the Decision Tree algorithm in our explanation.

The goal of decision trees is to find criteria with which to divide the data, such that these divisions explain all possible outcomes of the target. In other words, when the data is divided based on these criteria, it shows the cutoff points within the features that influence the outcome of the target. For example, if a Decision Tree were trained to predict if a student passes an exam, it may find that all students who studied less than 1 hour do not pass. These divisions that are recognized within the data are then “recorded” as a set of if-then-else rules in the form of the tree. To make decisions on new, unseen features, we simply follow the tree that has been created using the training data.

Let’s look at how this works using the training data shown above.

Decision tree for the task of predicting if someone plays golf

The Decision Tree machine learning algorithm starts by selecting a first column at random on which to split the data. Here, it has selected the column “Outlook” to start. It then splits the columns based on the values it can take. In our dataset, this column has 3 potential values it can take: sunny, rainy, and overcast. After this split, we see that one of the values, overcast, already fully represents one of the outcomes: Yes. This means that based on the training data, every time it was overcast, the person played golf. The algorithm does not need to split this node any further, as it already represents one complete criteria for explaining the relationships between the input features and whether or not the person plays golf. However, for the other 2 values of sunny and rainy, this first split does not provide us with a complete decision criteria. On sunny and rainy days, our dataset contains both instances where the person did play golf, and where they didn’t. This means we need to continue splitting in order to find a complete criteria. For the next split, the algorithm selects the column that will bring us a close as possible to a complete decision criteria. For the sunny subset of data, splitting again on the column “windy” provides a complete decision criteria: on sunny days where it was windy, the person never played golf, and when it wasn’t windy, they always did. The same is true after splitting the rainy subset on the column humidity. We know our Decision Tree is finished training when each of the terminal nodes, or “leaves”, contain only one value of the target.

In this example, that means that all terminal nodes contain either “Yes” or “No”. When this is the case, we have enough decision criteria that all potential situations can be accounted for and a prediction made for them.

After this happens and our tree is trained, we get predictions for new features by following the decision criteria of the tree. This process is shown below. Let’s assume we have new feature data for a new day. This day was not included in our training data. New data points that were not used to train our machine learning algorithm and develop our model, but rather that we make a prediction on using the trained model, are called test data.

Data for a new day to predict
Following the Decision Tree to get a prediction for our new datapoint.

Going through the tree that represents the relationship between the weather features and our target value of whether or not the person will play golf is how we arrive at our prediction, shown below.

Prediction on new, unseen test data by following the tree

Following this process, we can see how the decision that is ultimately made by our machine learning model is entirely dependent on the data used to train it. This means that it is also sensitive to the conditions surrounding the training data. In this case, that means that we cannot simply apply this trained decision tree to another golfer, who may well have different preferences. It also means that the model will likely not be applicable forever and will need to be updated if these conditions change. For example, what if the training data only includes data from the summer, when the golfer didn’t mind playing in the rain, but in the winter this no longer holds true and she only plays when it’s sunny or overcast? Additionally, we’ve seen through this example that our algorithm can only learn from data that we show it. It obviously cannot split on data that we have not given it as features. What if another feature that isn’t included in our dataset is actually the most important criteria on which to split the data, like whether or not it’s a weekend day? All of these factors impact the outcomes and predictive power of our machine learning model by fundamentally changing the shape of our resulting Decision Tree.

How bias can be “baked in” to machine learning models

In the last section, we see how a simple machine learning algorithm, a Decision Tree, is trained. Once it is trained using the training features provided, it creates a Decision Tree in the form of criteria that split the data and lead to a prediction for the target value.

In this section, we return to our example of developing an AI model to use for hiring to show how these inherent features of machine learning algorithms and the way they use their training data to infer a model can lead to biased predictions.

First, let’s explore how the process of hiring typically works if no AI is involved. Virtually all candidate hiring processes involve the candidate submitting an application, which always contains a CV or resume, and can also include supplemental materials like a cover letter or education certificates. Here, we will focus on the information contained in a CV.

Example CV

In a typical hiring process, an HR representative or hiring manager from the specific department will “screen” CVs, checking them over and generally spending less than ten seconds on each one. In this extremely short amount of time, the CV screener is of course not able to read everything. As mentioned before, human CV screeners rely on biases of their own when making up their mind about a CV in these first few seconds. They may look at the format and decide it looks unprofessional and filter out the candidate based on this. Furthermore, if one candidate includes information about things like their certifications or conferences they have presented at, this may become a new “criteria” in the CV reviewer’s mind and they may use that (either consciously or subconsciously) to filter out other candidates, even if such things were not originally factors that was meant to be considered in the hiring process for the role. Human CV reviewers may also bend the qualification rules for one aspect if a candidate seems to them to be particularly strong in another area.

After this initial phase of CV screening, a portion of the total candidates who applied are then passed on to the next round, where they interact with more humans through interview stages like in-person and phone interviews, case study challenges, and group process days.

If we want to train a machine learning algorithm to take over the task of hiring, we have to first decide which specific task in the hiring process it will manage. This is equivalent to defining what our target variable will be — what our machine learning model will be trained to predict. Generally, most AI systems for hiring focus on the first stage of the hiring process, resume screening, as it is generally seen as the easiest to automate given that there is far less human interaction than in the later interview stages. For this task, our target variable would be to predict if a resume or CV (and its associated candidate) should be passed on to the next round of interviews. This means it is a binary classification problem: we have only two potential outcomes in our target variable, either the resume is passed on, or it is not.

The next step in preparing to train a machine learning model to screen resumes is defining what our features will be. Just like in the golf playing example, we need to identify features that will help our model learn the relationship between these input features and the target. The better these features actually explain or proxy the data, the more accurate the subsequent predictions from our model will be.

These features of course also must be values for which we have data. In this case, this means our features must be information that is contained in the CV. For example, even if we think that including a feature like the references from a candidate’s former employer would be relevant, we can’t include them if we don’t have access to that data. Once we have identified the relevant features for which we have data, we do what is called pre-processing to get this data into a form that the machine learning algorithm can learn from. Take for example our golf dataset: here, we have processed the weather data into distinct rows from which the model can learn. Each row contained the same information, and was aggregated to the same timeframe of one day. For our resume reviewer model, we could want to aggregate the data not by day, but rather by candidate. Each candidate would be represented by one row in the dataset. We would then fill in a value for each feature corresponding to that candidate. So, to fill in the “Experience — Years” feature column for each candidate, we would need to add up the years the candidate spent in each job they had previously and put this number as that candidate’s value for the column. In this way, we extract the data contained in each candidate’s CV into a format that the machine can understand and learn from.

Once we have identified the relevant features in our training dataset, we need to identify how these feature sets correspond to our target. This process is called labeling. Labeling in this case means filling in the target value for each candidate. Recall that each candidate is represented by one row of data in the dataset. For each row, we would fill in the final column of “CV Passed?” with either the label “yes” or “no”, representing the 2 pathways that a CV can take after being reviewed. Labeling is crucial, as without it, our algorithm cannot learn how a given feature impacts the target output.

Training machine learning models always requires iteration. It is very common to start with an initial, small dataset with only a couple of columns of features that are easy to extract from the raw data (in this case, our raw data is the CVs). Then, based on how the model performs with this initial data, additional features are usually added later on to try to improve the model’s accuracy.

Let’s assume in this example we’ve taken just four features that are relatively easy to extract from the CVs and seem like basic attributes for describing a candidate: the person’s age, gender, degree (simply degree level), and years of experience. After labeling this dataset with whether or not the person’s application passed on to the next stage, we are left with the following dataset:

Example dataset — resume screening

Now that we have our first dataset with training features and target labels, we can train a Decision Tree model. Based on the training data we used when training the model, we are left with the following Decision Tree:

Resulting Decision Tree model from our training dataset

Now like we did in our golf example, we can follow the trained tree to get a prediction for a new datapoint:

Following the trained decision tree to get a prediction for the new datapoint
Prediction for the new datapoint

Based on our Decision Tree, the candidate does not pass. As this our first prediction with our newly trained hiring model, we want to look into the decision that the model made. The great thing about Decision Trees as opposed to some other machine learning methods like Neural Networks is that they give us a set of human-readable rules that explain why a decision was made. Looking at the decision rules, we see that the criteria that ultimately decided to exclude the candidate from the next round was their age. The fact that they were older than 60 is what made the model decide to throw the candidate out. What? How could this happen? How could it be that our brand new machine learning model for resume screening is filtering out a candidate with 15 years of experience and a master’s degree, simply based on his age? That certainly doesn’t seem unbiased!

The answer, of course, has to do with the training features that we selected for the model to learn from. Machine learning models do not learn causality. As we showed before in the golf example, they do not try to find the actual cause of a given event or of a certain value of the target variable. Rather, they simply split the data based on correlation. This is a very important distinction. Additionally, the models base their representation of the relationship between inputs and outputs solely on the past, labeled data. In all of our past labeled data that was used for training, there was no one over the age of 60 whose application moved on to the next round. Thus, the model learned from this pattern, and what’s even more concerning is that it will now apply that pattern to future hiring decisions.

In the Decision Tree for our hiring example, we also see other disturbing trends, like the cutoff under “bachelor degree” where the candidates are split based on gender — male candidates with a bachelor degree all got through to the next round, whereas female candidates with a bachelor degree were all rejected. It’s easy to see from our Decision Tree that the algorithm identified this pattern, and now, worryingly, has saved it and will apply this very biased thinking to new data. However, what we don’t see from the final Decision Tree, which is based off of just a few columns of data gleaned from the CVs, is if the labeled data actually contains such a causal bias, or if this is a spurious relationship simply found from by the model. For example, looking back at our initial data, we see that each of the female candidates with a bachelor’s degree all had less than 5 years of experience, whereas each of the male candidates with a bachelor’s degree had more than 5 years of experience. It could well be the case that the initial resume reviewers who screened the resumes and thus created the labels were not basing the decision based on gender at all, but rather simply based on a hard rule of 5 years of minimum experience. We also have no information in our current dataset about what those years of experience actually involved, nor do we have information about the area of study their bachelor’s degree was in. Including these features that should be used to make hiring decisions will help us drastically decrease the bias in our model. They could also help us understand the true criteria the labeller used when labeling the data initially.

Removing bias from the hiring system

It’s clear that the features we included in our original dataset led to a biased model that relied on features like gender and age to make decisions. When training any machine learning model, it is extremely important to think about which features to include, as the model will always try to find features that correlate with the target, not those that have a causal relationship with the target. If we don’t want our model to find a potential correlation between features like a candidate’s age and gender and whether or not they were hired, we should simply not include these features when training the model. Instead, we should explicitly use only features that we would want our model to use when making predictions on new data.

In the new dataset below, each row still represents the same candidate as before, in the original dataset. The new dataset differs only in the specific information that was gleaned from each CV. Rather than pulling out a candidate’s age and gender from their CV, we have instead pulled out the level of their last job, and the field of study from their academic degree. Their degree level and years of experience are the same as in the last dataset.

New training data showing different features for the same candidates

Now that we have new features, we can re-train our Decision Tree. Our new Decision Tree looks as follows:

New, “unbiased” Decision Tree

Since the Decision Tree no longer sees the candidates’ age and gender, it is forced to use other columns on which to split the data and derive the decision criteria. We see that based on this new Decision Tree, every candidate with a degree in a STEM field (science, technology, engineering, or math) passes and gets moved on to the next stage. Every candidate with a humanities degree or no degree is rejected and does not move on to the next stage. For candidates with a business degree, the Decision Tree algorithm learned that the person’s last job is correlated with whether or not they were passed on to the next round. If the person with a business degree had previously worked as a manager, they advanced to the next round. If not, they did not.

Now let’s look at how our new model would classify our test case from before:

Test case to predict on, with new data points for field and last job included
Following the Decision Tree for the new test case
Prediction for the test case following the newly trained Decision Tree

This new Decision Tree shows how a machine learning model can be even less biased than a human resume reviewer when making hiring decisions. In this example, the algorithm saw no information about features like the person’s age or gender, or if they studied at a prestigious university. As the resulting Decision Tree shows, it has based its decision criteria solely on features that are directly related to a person’s ability to perform the job, like what subject they specialized in at university and if they had prior management experience.

Avoiding bias

In this article, we’ve shown how crucial the input data features that are shown to a model are for avoiding bias. With the right features, a model can be even less biased than human decision makers whose decision making may be clouded by unconscious biases. With the wrong features, a machine learning model can reproduce biases that were present in the humans who labeled the data in the first place. As our hiring example also showed, they can also even find new biases that perhaps were not even there in the original labeling.

One of the best ways to reduce bias in machine learning models is to include relevant stakeholders in the feature selection process. For example, if an employee from HR who specializes in diversity and inclusion had been included in the feature selection stage, they likely could have spotted the problem with including features like age and gender before they created biased predictions. Another important step in preventing biased models from making biased predictions is to test them on a wide and representative array of test data. This can help identify biases by identifying areas where the model is underperforming, or gives an unexpected prediction.

Above all, it is crucial when training machine learning models for prediction that we understand how they learn. Knowing how machine learning models learn the relationship between input features and the target output prevents us from including data that can point to bias in our training data.

--

--

data4help

Helping non-profits and NGOs harness the power of their data. data4help.org