Trusted AI: Finding Bias in Your Data

by Sohom Paul, Associate Product Manager, Salesforce

Welcome to Trailhead University

Imagine a university, say Trailhead University, is using machine learning to optimize its student counselors’ workloads. Rather than have its counselors spend time convincing each and every student to accept their admissions offers, the university wants counselors to focus on the most likely ones. Now, Trailhead University can reinvest these productivity gains into personally reaching out to students who are likely to matriculate.

How would they go about building an accurate predictive model? To predict the likelihood of a student matriculating, Einstein needs to be trained on examples of both students who accepted their offers and students who declined their offers. Let’s assume the training data looks like this:

If the university isn’t careful with its “training data,” it may unintentionally discriminate against some of its prospective students. In other words, the model may be biased. As seen in our previous blog (Einstein Prediction Builder: Thinking Through Predictions with Bias in Mind), we define bias as: to wrongfully impose a relative disadvantage on persons based on their membership in some salient social group, e.g., race or gender. The bottom line is: If your model is trained on “biased” data, its results will also be “biased.”

Before we dive into the four causes of bias, spend some time analyzing the above dataset. Ask yourself questions like:

• “Which groups here are advantaged vs. disadvantaged?”
• “Does the distribution of matriculating vs. non-matriculating students look equitable across the advantaged and disadvantaged groups?”
• “How may utilizing training data like this impact my model’s outputs, and accordingly, my stakeholders?”

Now that you’ve answered some of these questions, you’ve already accomplished the first step (of many) towards building a fair predictive model!

Four Causes of Bias in Data

1. Sample Bias

First off, your training data must be representative of your broader population. E.g., If you’re striving for a diverse class, you need to start reaching out to more groups of students. Since your model has been trained on two types of students, it’s bound to make inaccurate (and perhaps unfair) predictions on types of students it’s never seen before. E.g., Students with disabilities and male students of color aren’t represented in your sample. Hence, you’ve fallen for sample bias.

2. Data Imbalance

Your data has 2 times more men than women, which is likely to skew the model’s outputs towards men. This is a common problem in machine learning because data is rarely balanced. E.g., If you’re making a prediction on the state of California, its most populated cities (like Los Angeles and San Francisco) are bound to dominate the dataset. While techniques like upsampling and downsampling may help balance the datasets, we recommend exploring ways (as in the case of Sample Bias) to improve your marketing, outreach, and messaging towards broader groups.

3. Treatment Disparity

Among men, 75% matriculate. Among women, 16.67% matriculate. Although that’s a staggering difference, Treatment Disparity doesn’t always imply bias. Perhaps one group is more qualified or has more opportunities to succeed than the other. That said, if you notice that men receive a much higher relative % of matriculation than women do, they should also have higher campus visitation rates and demonstrate better cultural fit to support that difference. When in doubt, compare both groups quantitatively and qualitatively. Chances are, huge treatment disparity is hard to explain and therefore implies bias.

Food for thought: Even if men can “explain” their higher relative matriculation rates (through higher campus visitation rates, for example), it’s important to consider that men may also have more opportunities for success than other disadvantaged groups. Thus, Trailhead University may choose to pursue disadvantaged groups at higher rates to achieve equity in the long-run. E.g., Affirmative Action.

4. Outcome Disparity

In the Treatment Disparity example, we see that 75% of men accept their offers, while 16.67% of women accept their offers. If there were an equal number of men and women, then men would matriculate 4.5 times more than women [math check: 0.75/0.1667 ≈ 4.5]. However, our dataset has twice as many men as women, which means that overall, 9 times more men than women matriculate! See how Data Imbalance can multiply the effects of Treatment Disparity to form Outcome Disparity? Outcome Disparity may, yet again, be explained by statistical differences, but it’s up to you to investigate how your business processes lead to these differences in the first place. At the end of the day, your models are training on this label [accepted offers vs. declined offers]. It’s relatively self-explanatory that the model will, by default, favor men over women in its predictions.

Watch out for Proxy Bias

Referring to the diagram below, what are sensitive fields? Coming back to our definition of bias, sensitive fields are personal attributes that define a person’s membership in some salient social group, e.g., ethnicity or gender. Now, what are related fields? Related fields serve as proxies (hence why we call this Proxy Bias) to sensitive fields, i.e., they are correlated to or are predictive of sensitive fields. Let’s see an example of this below:

What related fields can you derive from a person’s name?

1. Gender (common male first names: James, John, Robert; common female first names: Jennifer, Susan, Lisa).
2. Ethnicity (common Vietnamese last name: Nguyen; common Indian last name: Patel).
3. Religion (common Muslim last name: Khan).

Even if you remove sensitive fields like Gender, Ethnicity, and Religion but include the related field Name, bias will be captured subliminally in your models.

The same goes for common related fields like Salutation and Postal Code. However, sensitive and related fields aren’t limited to these basic ones we’ve listed here. They vary based on industries, use cases, and specialties. E.g. In Banking, a loan applicant’s source of income can drastically impact their probability of receiving a loan, but that may not be the case in an industry like Healthcare.

Now what?

So you’ve identified all the types of bias in your data. Now what? Can we flip a switch on our iPhones?

Maybe that’s too easy, how about some code?

import library fairness // use function makeDataFair()
fairData = makeDataFair(biasedData) // problem solved!

Unfortunately, this is a complex issue, and there is no simple, “magical” solution. Bias is a systemic problem that exists because we are imperfect humans living in an imperfect society. The process of bias mitigation is hard work, but it all starts with knowing whether a problem exists. So here’s our advice to you:

1. Start with the 5W’s of Data Quality and Accuracy (see Kathy Baxter’s framework here: Dirty Data or Bias Data?)
2. Follow up on each of these questions with data visualizations that uncover the biases we discussed here.
3. Take your findings with a grain of salt — again, just because you see treatment and outcome disparities doesn’t mean you necessarily have bias. While comparing each of your groups, question your findings (refer to Natalie Casey’s business expert questions: Einstein Prediction Builder: Thinking Through Predictions with Bias in Mind).
4. Once you find potential areas of bias, spend time introspecting on how your business processes impact your data.
5. Lastly, think of ways you can change your business processes to achieve parity across groups. Perhaps, run marketing campaigns to attract more disadvantaged groups to your application funnel. Or build new products that cater to those groups. At the end of the day, your models are only as fair as your data is. And your data is only as fair as the business processes which generate them.

Good luck. And remember, Einstein’s here to help you through these challenges!