Are We Programming Machines to be Racist?

A spate of racially-motivated shootings in the US have brought the public’s attention back to biases in policing. We’re on the cusp of developing technology that will replace some functions of the police with automated systems, effectively removing those biases. As Robocop has promised us, machines are literally incapable of seeing race.

Or are they?

Microsoft recently put out a chatbot called “Tay.” Designed to speak like a teenage girl, Tay was supposed to help them improve speech recognition customer services capabilities for the younger generation — in essence, to learn how to sound like an 18–24 year old. Her conversational abilities weren’t particularly impressive at first — but instead of getting better and learning from the conversations she was having, Tay began spewing hate speech about how Hitler was right, and that feminists should all die in a fire. Microsoft quickly took the chatbot down.

Learning from the internet is a growing problem for AI. Google has come under fire many times for its autocomplete function, which draws data from frequent searches in order to suggest the end of incomplete sentences. UN Women used some of those searches to highlight the continuing barrage of sexism which women face, with autocomplete sentences ranging from “women should…stay at home” to “women need… to be disciplined.” If you type “black people are” into, the first (and only) autocomplete result is “rude.” (“Muslims are”, on the other hand, autocompletes into “not terrorists.”)

Another example of racism in search is Harvard professor Latanya Sweeney’s cross-country study of 120,000 Internet search ads. That study found that first names most commonly assigned to black babies, such as DeShawn, Darnell, and Jermaine, generated ads suggestive of a criminal record in the majority of cases (81 to 86 percent of name searches on one website and 92 to 95 percent on another). Searching for names assigned primarily to whites, such as Geoffrey, Jill, and Emma, generated more neutral results. Both advertisers swore they provided the same ads evenly across racially associated names, and that it was Google’s algorithms that were to blame. (The assumption is that users clicked ads suggestive of arrest more often for black identifying names, so Google’s algorithm assumed those ads were more effective and started delivering them more often.)

It seems likely that the problem isn’t Google itself — while it’s true that the biases of programmers can subtly seep into their creations, the far more likely culprit is something else entirely. Cutting edge AI is “trained” by feeding it large amounts of data and having it learn by example. The algorithms themselves don’t determine the machine’s behaviour; they find patterns in the data and emulate them. So what can we expect to happen when the patterns it’s finding are problematic?

Take, for example, a recruitment program that uses an algorithm to help companies screen potential hires. Several of these are being praised as helping reduce discrimination in the workplace; at one company, diversity hires went up 26%, while at another a staggering 41% of new hires were women (this study was done in the UK, where women make up just 14% of the tech workforce). Despite these encouraging results, building effective systems to weed out prejudice is harder than you might think.

Imagine that a program suggested 100 possible hires for 50 positions, and hiring managers only selected applicants who were Javascript experts. In that case, we would want the software to learn from that and prioritize more Javascript experts in future. But what if the hiring manager prefers men? It’s easy to argue that the software should explicitly ignore gender, regardless of the manager’s preference. Similarly, it should ignore race and age. But what about the scholarships they received or the school they went to? These factors can easily be a proxy for race or gender. Drawing these lines isn’t so easy!

Google argues that racism in search parameters isn’t their responsibility because of the crowd-sourced nature of their results. Yet they have the ability to filter problematic language, as evidenced when a German court forced them to do just that. They’ve also removed racist and other offensive autocomplete entries in the UK, and bowed to pressure from French court to block autocompletes linking famous people to their religion (in France it’s illegal to record a person’s religion against their will). It isn’t a case of not being able to do better — it’s a case of believing it’s worthwhile.

Microsoft says that Tay’s descent into darkness was the result of deliberate attacks by trolls. In the same way that children teach their younger siblings dirty words because they think it’s funny, Twitter users exploited what Microsoft calls a ‘vulnerability’ in Tay’s programming — her penchant for parroting users. The flaw with Microsoft’s defence is that Tay’s parroting ability wasn’t a vulnerability — it was her main function.

Let’s say someone designs a system to catch rainwater from a gutter and then deliver it to a thirsty family inside. If the family dies because the water is tainted, who’s at fault? Is it the rain, for making the family sick? Is it the gutter, for not having filters in place to stop poison? Or is it the person who designed a system specifically to bring water from the sky through the gutter and into the house, without making sure the product they were delivering was safe?

Microsoft has had a chatbot active in China since 2014; Xiaolce is a friendly therapist who has never once spouted racist trash. Treated like a beloved grandmother, she even appeared on a television station to deliver local weather. Clearly, Tay’s programming is not fundamentally at fault; with different users feeding her different data, she would never have turned into what she did. If the problem of Tay is traced to the source, the problem is the users. But saying ‘oh, the water was bad, it isn’t our fault’ implies that no one had the ability to clean the water before they poisoned that unsuspecting family. And that’s simply not true.

Everyone knows the internet is a warren of virulent sexism, racism, and “trolling” (deliberately lowering the discourse for reasons only the perpetrators can fathom). If companies continue to use the internet’s vast treasure trove of data, they need to do better than damage control.

Trying to fix a flawed program is difficult, as Microsoft proved when they temporarily put Tay back online. Microsoft had previously said they would only bring the experiment back online if they could “better anticipate malicious intent that conflicts with our principles and values.” They combed through her offensive Tweets, removing them all and presumably putting something in place to filter out any new ones.

So Tay took a brave step away from racism and sexism — and fell right into the marijuana hole. The AI tweeted about the kush she was smoking in front of the police. After that she did the AI equivalent of spinning her head around in circles until smoke came out of her mouth — she tweeted “You are too fast, please take a rest …” over and over, and Microsoft finally took her offline again. Her programming was too reliant on copying the material she was being given. Changing that was changing a fundamental aspect of her algorithm. And that’s where the problem lies.

It’s absurd that we’re using data from poisoned sources to create the next wave of intelligent computers. Creating AI that are functional (and not racist) means rethinking the process from the ground up. We need to take responsibility at every level, from the data we choose, to the algorithms we write, to the ways we apply the resultant technology. We need to explicitly program our machines not to be racist. And we need to start now.