The origins of bias and how AI may be the answer to ending its reign
Have you ever had a moment when you realized something you thought was so obviously true, suddenly became so obviously false, and in that instant your whole understanding of the world changed?
I think of the day I realized my parents were actually fallible human beings. Up until then, part of my subconscious was convinced they were unquestionable demigods. It wasn’t until my 30’s that I started to really question the validity of my opinions and recognize that I was biased towards their viewpoints—both on the world and about me. I gave more weight to their ideas on politics, religion, morals, and even my own capabilities and characteristics than I did to the opinions or facts presented by my experiences. That was the moment I began making it a point to think harder. To examine why I believed something and attempt to form less parentally biased opinions of my own.
Moments of enlightenment, realizations, epiphanies–they feel so profound that the rhythms of their re-tellings have become classic tropes in inspirational storytelling, motivational speaking, and probably the last good TED talk you saw. Often they are awakenings to our implicit biases.
Implicit bias refers to the patterns learned by our brains from the small number of examples, experiences, stories, etc. we’ve been exposed to. These patterns allow us to make useful predictions to keep ourselves safe—e.g. who to trust—but also introduce unconscious assumptions that may not be true for a specific individual we meet.
Not all implicit biases are entirely bad. In fact, without those instincts to protect us, we may not be here today. Things like telling a child to not trust strangers isn’t necessarily comprehensible to them, but we drill it into their brains until it’s an unconscious action that protects them. I can attest to relying on these instincts heavily as a woman who does a fair amount of solo traveling in foreign countries. I am certain they’ve kept me safe from dangerous situations. But those same instincts can lead to fear and discrimination. They’re challenging to recognize and overcome because they’ve been hardwired into our brains for millennia.
According to cognitive neuroscientists, we are conscious of only about 5% of our cognitive activity. Most of our decisions, actions, emotions, and behaviors depend on the 95% of brain activity that happens in our subconscious.
Without realizing it, we are driven by implicit biases that we continue to struggle in recognizing and more so, to overcome, even though they’ve caused thousands of years of pain and suffering.
Ironically, it’s the technology we fear will have bias towards us, artificial intelligence, that has awakened us to and perhaps will help us to end the negative effects of this remnant from our evolution out of the wilderness.
AI is a reflection of its creator
To understand how bias is becoming embedded into AI, we have to look at how we build it.
To start, AI creators design these experiences by forming the questions the AI will solve for and deciding how it will do so. Then they fuel the AI with logic (models) and knowledge (data). In return, the AI churns out results, responses, and insights based on the creator’s design decisions and the structured knowledge it’s been provided. Because the AI is applying it’s logic to vast amounts of data, the biases within the design, the logic, and the knowledge become magnified. Suddenly we’re looking at a giant projection of a trait we didn’t even know we had.
AI, most often in the form of machine learning models due to their reliance on big data, reveals our biases through the patterns of interaction and forms of discrimination that we embed into it because we are unable to consistently be conscious of or eradicate our own implicit biases as we are creating these systems.
Biased AI results are not just a reflection of who we are today, but of everything that’s led up to this cultural state since the beginning of history–determining who lives in safe neighborhoods, who can afford education, who gets good jobs.
As well as the conscious and unconscious discriminations AI creators embed into their designs, because minorities make up the largest percentage of low income households they often don’t have access to the internet, devices and apps necessary for contributing to the big data that’s fueling our AI systems.
Be it by race, gender, culture, age, or religion—all minorities have been underrepresented in jobs in the technology industry since we began building AI back in the 1940's.
This lack of diversity is how Google ended up with their infamous photo search fail of 2015, when the machine learning models that power their search engine resulted in photos of African Americans when prompted to find images of gorillas.
This specific instance happened not because designers and developers were actively trying to reinforce racial slurs, but because they probably started training their machine learning models with their own data. Because most of Google’s developers are Caucasian and Asian, it’s likely they weren’t using photos that represented a wide diversity of races, or even thinking about potential biases as they were under the gun to achieve acceptable accuracy for every possible search instance.
Unlike the Google example, there are examples of bias explicitly being embedded into AI, like this glaringly obvious one that we never seem to talk about. Can you match the masculine and feminine names of our most popular AI assistants to the types of tasks we’ve assigned them?
Choice A: Watson/Einstein
Choice B: Alexa/Siri
Question 1: ___(insert choice here)___, call me a taxi.
Question 2: ___(insert choice here)___, review all customer complaints coming in from Asian countries for the past three months and tell me the top five issues we need to address.
We also see it happening in the news when journalists purposefully manipulate the way their audience interprets their data based on how they visualize or articulate the results in order to tell a story that reflects their own biases rather than objective facts.
Race and gender are setting off everyone’s bias alarms at the moment, but don’t forget, humans have endless possibilities for expressing our “me vs. you” instincts. Young vs. old, straight vs. gay, hearing vs. deaf. If you want to go deeper into understanding cases of bias in AI, this book by Cathy O’Neil is essential reading:
What’s being done
Scientists have broken down why it’s hard for humans to overcome bias into a few simple reasons:
- We’re swayed by anecdotes
- We’re overconfident in our own knowledge
- We’re biased by our prior beliefs
- We’re seduced by graphs, formulas and scientific speak
- We can be conscious of our biases, but we can’t entirely prevent them from effecting our decisions
And I have my own theory for why AI will be what finally helps us overcome it.
- Humans are biased, and that bias has caused a lot of trauma in the world, but they haven’t been able to overcome it because it’s hardwired into their brains.
- Humans created AI, and suddenly were fearful that robots will be biased against them.
- Humans want to fix the bias in AI, but they can’t easily recognize or prevent embedding their own implicit biases into the systems they create.
- AI is a tool for recognizing things in vast amounts of data.
- Therefore, while AI is the thing we’re trying to remove bias from, it’s also the tool we can use to recognize bias in the systems we build, as well as in ourselves.
Given this hypothesis, how are we trying to solve for bias in AI by using AI? Well, you’ll be pleased to know that the engineers, researchers and scientists of the world are hard at work. Check out this tongue in cheek chart from a graduate seminar on fairness in machine learning on the number of papers released on bias in ML recently:
As of 2018, nearly every major tech company is working on new solutions.
Facebook, Amazon, Microsoft, Google and most recently IBM have announced open source tools to examine bias and fairness in trained models.
Not every solution is rooted in AI, some are nearly analog in their approach—meaning they’re things you can do without the assistance of a computer. But in general, all of the ideas can be summarized into a few categories:
1. Getting broader data
MIT Computer Science AI Lab (CSAIL) researchers have created a method to reduce bias in AI without reducing the accuracy of predictive results. The key, Sontag said, is often to get more data from underrepresented groups.
I’m not sure we needed one of the world’s top research facilities to figure this out for us, but hopefully this leads to companies investing to increase the diversity of their quantitative and qualitative research. The simplest approach to solving for bias issues is to begin by gathering diverse datasets that inform our machine learning models. Diverse data is the absolute starting point for creating AI that takes all of us into account.
2. Datasheets
This is an engineering method being repurposed for tracking data about the data. Specifically, what did this data measure and how was it measured.
So if you were to purchase a dataset, let’s say “State Marriage Rates 1990, 1995, 1999–2016”, the authors of that dataset would include a standardized set of metadata specs — i.e. a datasheet. This document captures aspects of the research that play into it’s contextual, ethical, and biased traits that anyone purchasing that data should be aware of before using it.
Microsoft has been doing a significant amount of work in this area and is developing a tool that can “detect bias in artificial intelligence algorithms with the goal of helping businesses use AI without running the risk of discriminating against certain people.”
3. Bias auditing
These are tools that review the mechanics of machine learning algorithms and present reports that outline the levels of consideration various aspects of the data are being given and how they are effecting the results. Users can then test pre-configured solutions to see how they might improve their fairness ratios and decide which ones to apply.
One of the most well known examples of this at the moment is a tool by Pymetrics called Audit AI, which can examine the output of any machine learning technique. Audit AI is designed to detect bias, primarily in HR tools, but removal of any imbalance in an algorithm is up to the creator.
Facebook announced back in May 2018 that it was working on a similar system called Fairness Flow that can measure potential biases for or against particular groups of people. Given their track record for the past 12 months, we’ll see how that goes.
In September 2018, Google debuted their What-If Tool. It allows users to generate visualizations that explore the impact of algorithmic tweaks and adjustments to bias in their datasets on the fly. Very cool.
Similarly, IBM Research, released AI Fairness 360 also in September of 2018. It allows users to dynamically compare and contrast nine different aspects of bias correction to a dataset and determine the best out-of-the-box solution for improving bias and fairness.
Also very cool, but I might be biased.
4. Digital decisioning platforms for bias and ethics
Decisioning platforms help users quickly make decisions about how to improve their models for bias by applying machine learning models paired with natural language to provide automated insights and recommendations from the AI to the user.
They’re different from bias auditing tools in that they go beyond just reporting results. Their outputs are designed specifically as conversational recommendations and will potentially automate the application of any approved suggestions rather than requiring the user to apply the changes.
5. Conversational discovery assistants
IBM’s Project Debater is the first AI system that can debate humans on complex topics. It digests massive texts, constructs a well-structured speech on a given topic, delivers it with clarity and purpose, and rebuts its opponent. Eventually, Project Debater will help people reason by providing compelling, evidence-based arguments and limiting the influence of emotion, bias, or ambiguity.
In a category all on it’s own, IBM’s Project Debater has plans for one day being used to help us identify bias in our own conversations, designs, and perspectives rather than just policing AI systems.
Picture you’re working on solving a problem and you think you’ve considered every angle of the issue, but just to be certain, you turn to the AI console on your desk and ask it to debate the idea with you. Within minutes the AI responds with counterpoints to your hypothesis, offering perspectives that can be specific to identifying potential biases you hadn’t thought to consider.
This is the dream scenario, a bit like having your own Commander Data from StarTrek. So far, the first public debates have been impressive (see video below). You can read a full explanation of the technology behind Project Debater on the IBM Research website.
6. Design with intention
While all of these solutions demonstrate much promise for mitigating the risks of embedding bias into the data, models and interactions we create, it is not enough to rely on data scientists, algorithms, and AI itself to catch every instance of bias within a system, or more importantly, to know what to do about it.
Bias originates in humans, and therefore, if we really want to be in control of it, we must learn to become aware of it in ourselves before we can create machines that know how to deal with it for us. To begin with, we need to find new practices that help us recognize and consciously consider it’s causes and effects in our daily lives and in the systems we build.
At IBM, I’ve been researching and testing methods for integrating bias mitigation into traditional design thinking practices
I collaborate with accessibility and ethics experts to incorporate awareness practices into AI specific design thinking methods that teams now use to strategically implement AI into their products.I began this work three years ago, with a deep exploration into the components of human communication, which led to the Human-to-Machine Communication Model. This model is a map of the sequence of steps a computer needs to take in order to deliver cognitive experiences to human users—meaning experiences in which the computer can understand, reason and learn. It allowed us to recognize and simplify what designers need to be considering when creating cognitive AI experiences.
I first started testing this with teams by using the model itself as an exercise to help them find valuable uses for AI in their products. But, it quickly became apparent that it was too conceptual to be effective as a tool for guiding innovation. So, I tried a different approach, staring with something people were already familiar with — design thinking — weaving the necessary aspects of cognition the model identifies into well known design thinking exercises.
The first thing you learn in design thinking is to start by focusing on the user. So I started by looking into exercises for defining individual user personas and began adapting them to incorporate cognitive capabilities, capturing points that would take advantage of AI’s ability to deliver contextually personalized, anticipatory, conversational experiences.
This changed the usual method of mapping multiple personas into creating primary profile traits. Profile traits tell us what categories of personalization all users might need from the AI. This includes considering the nature of the environments users work in, the roles they play at work and in life, their professional and personal motivations when using the system, and identifying any outlying, or what I call exceptional, profile traits.
Exceptional profile traits describe the insights minority users could contribute to improve the system. For example, a product could have unique individuals who use it on multiple devices, or beta testers with expertise in UX, or anyone whose different view of the world could provide rare insights, such as those with physical impairments or unique dialects. The exercise helps teams identify and prioritize data sources they might not otherwise consider, giving them a chance to improve the experience for everyone through anomalous insights.
Most importantly, it gets teams to consider the new opportunities AI opens the door for us to create.
There was also the need for a few entirely new design thinking methods. The Data exercise for example (see image below) came out of an aspect uniquely important to AI systems—input. Input encompasses all of the sources of data sources the AI system could ingest, from Wikipedia to sensors, wearable devices to private databases. The question the exercise answers is, what sources of data could add value to your AI experience and what risks will you take on if you decide to use this data? A technical, design, and ethical question, it requires the full team’s input — including research, design, development, and strategy. The exercise breaks the question down into three categories:
- What data could we use from the world (free and purchasable data available to the public)?
- What data could we use from the system (private data from the owner or creator of the system)?
- What data could we use from the user (both public and private)?
We then look at the risks versus the value using the data would incur. By identifying early on what challenges might arise from these choices, teams are building a plan of action for security, privacy, bias, and ethics before they even get to ideation.
Consider a team that chooses to use research data gathered by their company to personalize an AI system that helps HR departments screen new job applicants. There are at least two points they’ll need to investigate before using this data that the exercise should capture:
- Will the way they envision using the data require permission from the individuals who contributed to the data? If it does, will the value the data provides offset the security risks that using this private data will create?
- What bias might be embedded into the data based who contributed and how it was collected? If the research only included people between the ages of 25–30, then this AI will likely be biased against anyone outside of that age range and could potentially prevent other qualified candidates from applying.
To date, this work has resulted in a sequence of eight design thinking phases. I test and teach this approach through the AI Camps I now facilitate for IBM teams around the world, as well as through my university courses. The outcome is a user-focused vision for AI expressed through storyboards, demos, dummy press releases, and a report on what value the AI vision will deliver and what will be required to build it — all rooted in technical feasibility feedback from IBM’s machine learning experts.
I’ve found that when corporate teams or students are beginning to innovate with AI, they’re mainly excited about the technology and design opportunities. They’re not thinking about the layers of cognition they need to consider or the consequences of their decisions. By using AI attuned exercises to consider bias, ethics, accessibility and privacy from Phase 1: The User, all the way through to Phase 8: The Project Brief, we can make designing for these issues an everyday part of development practices, not something only considered if there’s time — or worse, only after something goes wrong.
In the end, AI Camps are about helping teams deliver a strategic, fully vetted AI vision that answers the executive’s question, “What’s the most valuable way we should use AI in this product and what’s it going to cost me?”. But they’re also about bringing the realities of using AI to the front of the conversation rather than blindly charging ahead.
The stakes surrounding bias, ethics, accessibility and privacy are much higher now that AI is in the picture.
We can’t continue to put these issues on the back burner or we’re going to end up with a much larger and more costly problem on our hands.
The virtuous human–AI–human cycle
That was a lot to take in, I know. But now we can bring it all back to the fun stuff, the future of humanity! So without further ado, the story of my personal experience and perspective on AI and bias:
In 2015, I decided to turn my career towards AI because I have this grandiose, buddhist-like hope for what cognitive machines might help us achieve one day. I believe AI is the first opportunity humanity has had to bring empathy, understanding and equality to all life on this planet. What could that mean for us, the Earth, and any other intelligent lifeforms we may one day encounter?
Imagine three world leaders, conversing hundreds of years from now, each with a personal, lifelong AI system embedded their brains. As one leader speaks, the others see and hear insights about the speaker that bring layers of understanding and empathy through visual playbacks, history and behavior analysis, even recommendations on how to respond in a way that most clearly communicates and connects with that individual.
What would that world be like? One where we can actually understand each other, not get hung up on misunderstandings or our own experiences? Where we don’t turn to bigotry or violence or war because we can’t empathize with each other?
It’s a nice idea yes, but you’re right, this whole “future with machines” thing could go sideways on us so fast. As one of my colleagues pointed out, it’s just as likely that this brain embedded AI could be saying, “Don’t trust him. He is primarily interested in enriching his family at the expense of his own people. He does not respect your country.”
How we end up handling bias in AI poses one of the largest threats to the success of my utopian vision. Bias comes down to ethics and values—two things that don’t have a simple right or wrong answer. They’re imaginary laws created and enforced by humans, and we don’t all share the same opinions on what those laws should be.
Who’s going to determine what laws AI should abide by? The biggest nation? The richest corporation? The smartest hackers? What if we make AI that’s 100% politically correct, isn’t that perspective biased as well? What if we let AI prevent us from creating biased content or products or ideas? Won’t we all just become the same? One homogenized world with none of the beauty or inspiration that comes from our differences?
I didn’t think about the importance of bias when I had this Captain Picard inspired vision for AI. I was too excited by the potential of a new technology to take pause and consider the consequences. Not until I understood the mechanics behind creating cognitive experiences were my eyes finally opened to how small, unconsidered decisions can snowball into hate bots. Or poverty. Or preventing access to education. Or glass ceilings. It’s all the same cycle isn’t it, just different eras with different weapons for us to wield against each other.
My epiphany about tiny actions having big effects was the beginning of a waterfall of red-faced realizations for me, and my first move towards looking for ways to face and change my own biases, prejudices, and fears. I’m in the process of taking little steps, like following people on Instagram who don’t look or think like me. Volunteering in places where I’ll get to know people from different communities. Listening to podcasts that share first person perspectives on discrimination. I’ve started to find so many ways to broaden my tiny experience of the world. The more I learn, the more I want to see things change.
And of course, I’m far from being the first or only person looking for ways to overcome bias. Even as we’re seeing a resurgence of nationalism sweep across the globe, I also see incredible examples of people striving to find understanding and empathy for one another. One of my favorite examples is Daryl Davis, who is on a quest to reform the Ku Klux Klan’s hatred towards African Americans by building amicable relationships with their members through years of listening and conversation.
Despite knowing the odds are stacked against us, the thing that keeps me believing my vision could actually happen is my time spent with students. For the past year I’ve been teaching Advanced Design for AI at the University of Texas.
And this brings me to what I most want tell you about my experience with bias:
Issues of bias that effect my generation and the ones before mine are not the same for younger generations. Apologies for the biased generalization, but in my experience to date, I have found that younger people working towards their technology and design degrees tend to be more diverse, yes, but also more accepting, more open minded, and more caring than my generation is today. Some of this is a side effect of progress made before their time. Some I can’t help but attribute entirely to them.
They naturally gravitate towards seeing AI as a tool to understand, respect, and enjoy the differences in others. The students I work with have moved beyond old biases and are already creating new norms that are changing the world. I can see it in the projects they create:
- An app that helps people connect with other cultures by connecting new experiences while traveling to personal memories
- An AR experience for interactively improving public speaking skills that attunes its displays and recommendations to various cultural preferences and customs depending on where you’ll be giving your talk and who will be in the audience
- A healthcare assistant that focuses on the emotional needs of users often forgotten—people left waiting and worrying in hospital waiting rooms, and nurses at the front desk trying to take care patients’ families while attending to doctor demands
My hope is that they, and every generation after them, will build AI with continuous improvement towards equality and diversity. Just as I hope we’re doing better than the generations before us.
That doesn’t mean new prejudices won’t arise, or that we should wait for a perfect generation to appear before we continue this evolution with technology. My conclusion, after all of these observations and experiences I’ve just shared with you, is that now, more than ever, I regard this work as one of the most important issues we need focus on as technologists and designers. Not just in the AI we build, but in our understanding and awareness of our own minds.
It’s my belief that our best shot at getting the issue of bias right in AI is to use AI itself to help us evolve past our own biased instincts. For the first time in history we have a technology that is opening our eyes to who we are, is changing us as we speak, and could allow us to play a conscious role in who we want to become.
Dedicated with love to my husband who is always challenging me to see the world as he sees me, with open eyes and an open heart.
Jennifer Sukis is a Design Principal for AI and Machine Learning at IBM, based in Austin, TX. The above article is personal and does not necessarily represent IBM’s positions, strategies or opinions.
