Algorithms can leave anyone biased, including you
Part 2 of a 3 Part series — The Perils and Promise of Data
In the last article, we showed what happens when data strip emotions out of an action
In Part 1 of this series, we argued that data can turn anyone into a psychopath, and though that’s an extreme way of looking at things, it holds a certain amount of truth.
It’s natural to cheer at a newspaper headline proclaiming the downfall of a distant enemy stronghold, but is it ok the cheer while actually watching thousands of civilians inside that city die gruesome deaths?
No, it’s not.
But at the same time―if you cheer the headline showing a distant military victory, it means you’re a human, and not necessarily a psychopath.
The abstracted data of that headline strips the emotional currency of the event, and induces a psychopathic response from you.
That’s what headlines do, and they can induce a callous response from most anyone.
So if data can induce a state of momentary psychopathy, what happens when you combine data and algorithms?
Data can’t feel, and algorithms can’t feel either.
Is that a state of unfeeling multiplied by two?
Or is it a state of unfeeling squared?
Whatever the case, let’s not talk about the momentary psychopathy abetted by these unfeeling elements.
Let’s talk about bias.
Because if left unchecked, unfeeling algorithms can and will lead anyone into a state of bias, including you.
But before we try to understand algorithmic bias, we must take a moment to recognize how much we don’t understand our own algorithms.
Yes, humanity makes algorithms, and humanity relies upon them countless times every day, but we don’t understand them.
We no longer understand our own algorithms, no matter how much we think we do
At a high, high level, we could conceive of an algorithmic process as having three parts―an Input, the algorithm itself, and an Outcome.
But we are now far, far away from human-understandable algorithms like the Sieve of Eratosthenes, and though the above image might be great for an Introduction to Algorithms class―today’s algorithms can no longer be adequately described by the above three parts alone.
The tech writer Franklin Foer describes one of the reasons for this in his book World Without Mind: The Existential Threat of Big Tech―
Perhaps Facebook no longer fully understands its own tangle of algorithms — the code, all sixty million lines of it, is a palimpsest, where engineers add layer upon layer of new commands. (This is hardly a condition unique to Facebook. The Cornell University computer scientist Jon Kleinberg cowrote an essay that argued, “We have, perhaps for the first time ever, built machines we do not understand. . . . At some deep level we don’t even really understand how they’re producing the behavior we observe. This is the essence of their incomprehensibility.” What’s striking is that the “we” in that sentence refers to the creators of code.)
At the very least, the algorithmic codes that run our lives are palimpsests―documents that are originally written by one group of people, and then written over by another group, and then a third, and then a fourth―until there is no one expert on the code itself, or perhaps even one person who understands it.
And these algorithmic palimpsests are millions of lines of code long, or even billions.
Remember Mark Zuckerberg’s 2018 testimony before Congress?
That was the testimony of an individual who didn’t have the faintest understanding about 99% of Facebook’s inner workings.
Because no one does.
Larry Page and Sergey Brin don’t understand Google as a whole.
Because no one does.
And the algorithms that define our daily lives?
No one understands them completely, nor does anyone understand the massive amounts of data that they take in.
So let’s update our algorithm diagram. We need to understand that there are more Inputs than we can understand, and that the algorithms themselves are black boxes.
So here is a slightly more accurate, yet still high-level view of what is happening with our algorithms.
Again―there are more Inputs than we can understand, going into a black-box algorithm we do not fully understand.
And this can lead to many things, including bias.
A case study in algorithmic bias―a company is told to favor lacrosse players named Jared
A company recently ran a hiring algorithm, and the intent of the algorithm was to eliminate bias in hiring processes.
The algorithm’s purpose was to find the best candidates.
The company entered some training data into the algorithm based on past successful candidates, and then ran the algorithm again with a current group of candidates.
The algorithm, among other things, favored candidates named Jared that played lacrosse.
The algorithmic Output was biased, but not in the way anyone expected.
How could this have happened?
Algorithms are not compassionate, let alone sentient―but they are really good at finding patterns
In the above case, the algorithm found a pattern within the training data that lacrosse players named Jared tend to be good hires.
That’s a biased recommendation of course, and a faulty one.
Why did it occur?
Well, beyond us recognizing that we don’t understand the algorithm itself, we can cite thinkers like Dr. Nicol Turner Lee of Brookings, who explained on Noah Feldman’s Deep Background podcast that external sources of algorithmic bias are often manifold.
There might be bias in the training data, and quite often the data scientists who made the algorithm might be of a homogenous group, which might in turn encourage the algorithm to suggest the hiring of more candidates like themselves.
And of course, there is societal and systemic bias, which will inevitably work its way into an unfeeling, pattern-recognizing algorithm.
So to update our algorithm chart once again―
There are faint echoes of Jared and lacrosse somewhere in the Inputs, and we certainly see them in the Outputs.
Of course, both the full scope of the Inputs and the algorithm itself remain a mystery.
The only thing we know for sure is that if your name is Jared, and you played lacrosse, you will have an advantage.
This was a humorous example―but what happens when the stakes are higher?
Hiring algorithms are relatively low stakes in the grand scheme of things, especially considering that virtually any rational company would take steps to eliminate a penchant for lacrosse-playing Jareds from their hiring processes as soon as they could.
But what if the algorithm is meant to set credit rates?
What if the algorithm is meant to determine a bail amount?
What if this algorithm leads to a jail term for someone who should have been sent home instead?
If you are spending the night in jail only because your name isn’t Jared and you didn’t play lacrosse, your plight is no longer a humorous cautionary tale.
And when considering Outcome of a single unwarranted night in jail, there is one conclusion―
An Outcome like that cannot be.
Even if a robotic algorithm leads to 100 just verdicts in a row, if the 101st leads to an unjust jail sentence, that cannot be.
There are protections against this of course―the legal system understands, in theory at least, that an unjust sentence cannot be.
But we’re dealing with algorithms here, and they often operate at a level far beyond our understanding of what can and cannot be.
A brief aside — Algorithms cannot technically show bias based on Constitutionally protected classes, but they often find ways to do this
It’s not just morality prohibiting bias in high stakes algorithmic decisions, it’s the Constitution.
Algorithms are prohibited from showing bias―or preferences―based on ethnicity, gender, sexual orientation and many other things.
Those cannot be a factor, due to them being a Constitutionally-protected class.
But what about secondary characteristics that imply any of the above?
Again, algorithms are great at finding patterns, and even if they are told to ignore certain categories, they can―and will―find patterns that act as substitute for those categories.
Consider these questions―
- What gender has a name like Jared?
- What kind of background suggests that a person played lacrosse in high school?
And going a bit further―
- What is implied by the zip code of the subject’s home address?
So no, an algorithm―particularly one born of a public institution like a courthouse―cannot show bias against Constitutionally-protected classes.
But it might, and probably will if we are not vigilant.
Can algorithms make you biased? Considering algorithms are everywhere―the answer may be yes.
You don’t have to be an HR person at a Tech company or a bail-setting judge to become biased by algorithms.
If you live in the modern world and―
Engage in Social Media, read a news feed, go onto dating apps, or do just about anything online―that bias will be sent down to you.
Bias will influence the friends you choose, the beliefs you have, the people you date and everything else.
The average smartphone user engages with 9 apps per day, and spends about 2 hours and 15 minutes per day interacting with them.
And what are the inner-workings of these apps?
That’s a mystery to the user.
What are the inner-workings of the algorithms inside these apps?
The inner-workings of the apps are a black box to both the user and the company that designed them.
And of course, the constant stream of algorithmic data can lead to the perpetuation of insidious, and often unseen systemic bias
Dr. Lee gave this example on the podcast―
One thing for example I think we say in the paper which I think is just profound is that as an African-American who may be served more higher-interest credit card rates, what if I see that ad come through, and I click it just because I’m interested to see why I’m getting this ad, automatically I will be served similar ads, right? So it automatically places me in that high credit risk category. The challenge that we’re having now, Noah, is that as an individual consumer I have no way of recurating what my identity is.
Dr. Lee has a Doctorate and is a Senior Fellow at a prestigious institute, and has accomplished countless other things.
But if an algorithm sends her an ad for a high-interest credit card because of her profile, and she inadvertently clicks an ad, or even just hovers her mouse over an ad, that action is registered and added to her profile.
And then her credit is dinged, because another algorithm sees her as the type of person who clicks or hovers on ads for high-interest credit card rates.
And of course, if an algorithm sees that lacrosse-playing Jareds should be served ads for Individual Retirement Accounts, that may lead to a different Outcome.
Dr. Lee makes the point that this is no one’s fault per se, but systemic bias can certain show up.
Every response you make to a biased algorithm is added to your profile, even if the addition is antithetical to your true profile.
And of course there is no way that any of us can know what our profile is, let alone recurate it.
So individuals and the system are unintentionally biased by algorithms―what do we do?
First of all, we don’t scrap the whole system.
Algorithms can make you biased, and as I showed in Part 1, data can lead you to a form of psychopathy.
But algorithms and data also improve our lives in countless other ways. They can cure diseases and control epidemics. They can improve test scores of the children from underserved communities.
Rockford, Illinois employed data and algorithms to end homelessness in their city.
They solved homelessness, and that is incredible.
So what do we do?
We tweak the system, and we tweak our own approach to it.
And we’ll do that in Part 3.
Stay tuned!
This article is Part 2 of a 3 Part series — The Perils and Promise of Data
Part 1 of this series is here— 3 ways data can turn anyone into a psychopath, including you
Part 3 of this series — Coming Soon!
Jonathan Maas has a few books on Amazon, and you can contact him through Medium, or Goodreads.com/JMaas .
