MLFailures: Identifying Bias in Machine Learning Algorithms, and Who Gets to Do So

Machine learning algorithms — algorithms that make judgments on prior examples — are increasingly used to guide human decision-making. Judges use algorithms to determine sentencing in criminal cases. Insurance providers use algorithms to set rates. Lenders use algorithms to determine who deserves a loan (or not).

Machine learning (ML) is assumed to improve processes, making them more objective. In fact, the data these algorithms rely on reflect underlying biases around race, gender, age, and other factors. When trained on biased data, ML algorithms reflect and perpetuate these biases.

The Daylight Security Research Lab, housed in UC Berkeley’s Center for Long-Term Cybersecurity, has been working for almost a year on a project called “MLfailures,” a series of hands-on Python labs aimed at teaching students to identify (and address) bias, fairness, and security issues in machine learning models.

While these labs are currently designed for tech-literate students, they also have the potential to help bring in a new community of practitioners: people who want to address systemic social justice challenges — including racism and gender inequality — by uncovering algorithmic bias in working systems.

About MLfailures

The MLfailures labs teach students to understand how bias emerges in algorithms by examining real-life examples such as healthcare or lending.

Machine learning algorithms “learn” from past examples to produce future determinations. In contexts like healthcare, these data are shaped by biases in the institutions that produce them. For example, racial disparities in healthcare produce racial disparities in data about healthcare. The algorithms that learn from these examples frequently reproduce those disparities.

It’s easy enough to make a biased algorithm, but who is responsible for detecting this bias? Who are the “ethical hackers” uncovering bias in machine learning algorithms?

Currently, there is no clear pathway to train “bias-hunters.” The MLfailures labs represent a first step into building such a pathway: they teach students how to uncover (and ameliorate) bias in real-world algorithms. These labs are not only technical. They also ask students to consider the social context that surrounds building and deploying machine learning algorithms: where particular applications are deployed, by whom, who establishes that they “work,” and how.

How it Works

Take our healthcare lab as an example. In this lab, based on Ziad Obermeyer’s work, students encounter an algorithm that predicts a patient’s health risk. The output is a “risk score” assigned to each patient. The purpose of this risk score is to identify patients who need more care resources.

Among the inputs is how much a patient spends on medical care (health expenditure). The assumption underlying this model is that patients who spend more on healthcare have more severe health needs, and therefore should receive a higher risk score.

Students break down health expenditure by race. Black and white patients seem to spend about the same amount on healthcare, so the algorithm must be fair. Right?

Wrong. As students further break down the analysis (which is based on real data that has been anonymized and protected), they notice that Black patients in the model have more markers of chronic illness than white patients who were given the same risk score. In other words, the algorithm undervalues the health risk of Black patients relative to white patients.

But the lab does not stop at this technical observation. It asks students to consider why this bias occurs. Perhaps Black patients have unequal access to medical care, resulting in healthcare systems spending less on Black patients than on white patients, even when Black patients are more at risk. The lab then asks students to consider who designed the algorithm, who had the ability to determine that it worked correctly or was fair. These labs not only teach students how to identify bias, but also to consider how bias slipped into the algorithm in the first place.

Expanding our Reach

When we set out about designing these labs, we targeted undergraduate and graduate students at higher-education institutions in the United States. Since we’re based at UC Berkeley, we knew we’d be testing the materials on Berkeley students first.

Which students? When we began, we assumed many would be going through primarily technical coursework, earning degrees in computer science, statistics, and related fields.

Our motivation in reaching these students was simply that we believed they needed the material the most. It’s perfectly possible to graduate from UC Berkeley ready to start as a data scientist, even an ML specialist, without learning about bias or fairness in ML in a concerted way. As a result, many of the recent graduates who build machine learning models in industry may not know how to evaluate the fairness of a model they’ve trained.

MLfailures aims to reach these students. [1, 2] Of course, we hope these students are just the start. We intend for these notebooks to find their way into courses — technical and otherwise — across the country and world. Since the labs are creative commons licensed, instructors can modify them to their needs.

Beyond Undergraduates

This model of teaching undergraduate students has a limitation: it is fundamentally a model where those in technical roles design solutions for those who are not. Do we want the same group that creates these models to be the only ones “fixing” their harms? Who are we, as technical practitioners, to decide what those harms are, anyway? As Elizabeth Anne Watkins argues, “safety and security have different valences for different communities.”

A complementary model is helping underprivileged communities identify issues for themselves. The Algorithmic Justice League, Black in AI and Data 4 Black Lives all follow this model.

How can MLfailures best support their work?

A few weeks ago, I had a conversation with Larry Whiteside Jr, the President of International Consortium of Minority Cybersecurity Professionals (ICMCP), an organization that helps women and people of color who want to break into the cybersecurity job market learn skills and find jobs in an industry that is still one of the least diverse in tech.

I was hoping to get MLfailures into the hands of some students whose education is happening outside of four-year higher-ed institutions. He and I agreed that identifying ML bias is emerging as a key skillset for security and data science professionals. But, as he said, addressing ML fairness isn’t just a job skill. It’s a way for people of color to see work in security as something that directly benefits their communities.

A lightbulb went off for me. ICMCP helps women and people of color who want to break into the security field, but what about those people who don’t see security as appealing to them in the first place? After all, cybersecurity professionals overwhelmingly protect large corporations and shareholder value, which may not appeal to everyone.

MLFailures can connect security to issues in social justice. We can use these labs with people who may be vulnerable to machine learning bias — and who may be better-poised to help technical practitioners identify it. Who am I to say what counts as unfair or biased?

And not everyone who interacts with our labs needs to become a programmer, a data scientist, or a penetration tester. Activists can use these tools to build familiarity and awareness. Again, through this awareness, activists could observe new issues — issues that we, as designers, would not have identified.

We can use these labs with people who may be vulnerable to machine learning bias — and who may be better-poised to help technical practitioners identify it.

Back to our original question: who are the “ethical hackers” uncovering bias in machine learning algorithms? They should be members of the communities most affected by that bias.

To this end, we are reaching out to additional organizations to help broaden the reach of the MLfailures labs. As a start, we’re working with Brandie Nonnecke on integrating MLfailures into TechHive, a program aimed at high schoolers from diverse communities. And in the coming months, we will be working on reaching out to many more.

Exploring the how of making these relationships work is the difficult part. But it’s work we’re eager to do.

Are you a community organization that is interested in ML bias? Contact us! ffff at berkeley dot edu.

Bibliography

[10.1145/3334480.3375157] Fox, Khovanskaya, Crivellaro, Salehi, Dombrowski, Kulkarni, Irani & Forlizzi, Worker-Centered Design: Expanding HCI Methods for Supporting Labor, 1–8, in in: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, edited by Association for Computing Machinery (2020)

Footnotes

1. At the School of Information, we’ll be running a “Fairness Mini-Bootcamp” with our MLfailures notebooks in Josh Blumenstock’s INFO 251. For undergraduates, our healthcare notebook will appear in DATA 4AC, taught by Margarita Boenig-Liptsin.
2. There are excellent classes on ML bias and fairness for those who are interested; Moritz Hardt’s CS294, for example, or Ziad Obermeyer’s PH 196 (the latter was a huge help in putting together our materials). However, these courses are treated as “special interest” courses rather than requirements. Moritz and Ziad would probably agree that the high-level ideas from these courses should be required. They’d also probably agree that making “required materials” can be tricky; they need to be adaptable to a variety of contexts and skill levels.
3. I admittedly gave a pretty quick look through Casey Fiesler’s Tech Ethics Curricula spreadsheet to see if any tech ethics courses include work on labor, and did not immediately find anything. If you know of any tech ethics or AI fairness courses that include work on labor and organizing, please let me know: ffff at berkeley dot edu.