Transparent AI
Labelling Artificial Intelligence as a black box is not helping us, so we used design to show we can change this
Artificial Intelligence has transformed many aspects of our lives, becoming increasingly indispensable to the way almost every organization provides services and products. However, prejudice of different kinds (societal, race, gender etc.) creeps into machine learning operations too, which makes AI technologies not only interesting but also crucial to learn and ultimately question with a critical mindset.
At CLEVER°FRANKE, we’ve taken up the ambition to bring our expertise in data and design to create a comprehensive story on how AI operates and people could be better informed in daily life. Thanks to the ‘Pioneer’ grant awarded by the SIDN Fonds to support projects that aim to empower the user, we turned our vision into a solution by creating an interactive experience in order to:
- Allow people to understand AI based decisions;
- Bring the design practice for AI related topics beyond standard node / link diagrams;
- Demonstrate how companies can communicate about their training and use of AI;
- Enable laymen to discuss the implications or desirability of applications.
Identifying the problem: AI needs to be communicated differently
We started off by researching existing attempts to explain AI. The common problem that we identified was the lack of an engaging narrative that explains the process of machine learning. Most visual explainers are limited to node / link diagrams, constituting an unfriendly and convoluted look that doesn’t attract an uninformed public.
Besides, the portrayal of AI as a black box perpetuates a mystery attached to this technology and creates a belief that if only we could open the box, there would be a label saying that a machine learning program is discriminatory or biased. This assumption falls short because the content of the black box is not human. It’s not explainable simply by opening it up as it operates on a highly complex level at an incredible speed. In fact, the very fact that it has a knotty structure makes AI so strong and desirable, while compromises its understandability.
Making the entire AI process transparent
Instead of untangling the content of the black box, we aimed to put the black box in a wider context by looking at what’s happening around it and how it affects AI driven decisions. Our solution focused on making the following steps transparent:
- The bias in training data;
- The level of influence of each data point in training;
- The data that is being used when the AI starts operating in real-life;
- The level of influence of data entered by a user (and whether it differs from training).
We turned this complex system into a concrete experience by focussing on mortgage application scenarios so that it’s more relatable than node / link diagrams. Banks that use AI for deciding the eligibility of mortgage applicants firstly train historical data sets that belong to successful applicants. Based on this, AI makes inferences while operating.
Imagine an AI service that’s trained with certain salary ranges in order to say whether a salary is too low or sufficient enough for a desired mortgage amount. When a new customer makes an application and enters his salary, the AI starts operating, meaning that it uses the training data to decide based on the salaries of previous successful applicants who got that amount of mortgage.
However, there are cases where the training data is already installed with biases. While the data on the applicant’s salary is necessary, people fill out more information when making a mortgage application. For example, a postal code can be a way to assess the income of someone, as well as their religion and ethnicity. People living in certain postal code areas might have been more likely to get their loan refused in the past because of the prejudice of the bank. This type of a societal bias should not be mirrored by AI, yet there is no guarantee for that if the end-users don’t reflect critically on the data they provide.
The data that the applicant gives to the bank should also be transparent in terms of how much impact each data point has on the decision. If the data on salary and gender have the same level of influence on a mortgage decision, this might be an indication of bias and should be questioned by the applicant.
Sometimes, even if the data and the impact of each data point seem to have no bias during the training phase, AI might end up in biased decisions. For example, if an applicant has a special situation (or combination of situations) where his data is not represented in the training data, the AI might infer prejudiced decisions. When the level of influence between the training data points and real life data points is very different this is an indication of, for example, under-representation. In such cases, the applicant should be able to compare the results to the full process and have a basis for asking clarifying questions to the company involved.
A classroom experience
To make these processes (training, operation, the impact of data points) more engaging, we decided to blend our mortgage scenarios with a visual narrative. Based on our research, we developed the narrative that we crafted with certain mortgage scenarios into a click-through prototype — Transparent AI — that visualizes the full AI process. The prototype is designed to be used in a classroom environment to educate young adolescents (±16y/o), since they are growing up in an AI dominated world, yet there isn’t enough reliable information for them to develop critical thinking about AI technologies and their usage in everyday life.
Transparent AI consists of these main steps:
- Where do we use AI today?
- Why do we need AI?
- How does machine learning work?
- What can go wrong with a machine learning program?
After getting background information on AI, students can learn how it analyzes historical data sets and what goes into the training and testing phase to make decisions. During these phases, the prototype shows some cases where irrelevant data sneaks into the machine learning program like the applicant’s current postal code, which implicitly leads to biased decisions.
Questioning what data sets are required for what actions, and how training AI with these data sets can still cause it to operate differently, students can develop analytical thinking skills to gauge the fairness of AI programs.
While preparing this prototype for students, we realized that teaching about AI ultimately comes down to teaching about data. Since the AI is trained with a selected sample of data, that’s where the majority of bias slips in. However, teaching about data is not per se sufficient unless the services and products that people benefit from provide transparency in their use of data.
AI transparency in daily life
To further develop our scenario, we aimed to find a way to change the communication about the use of AI. Organizations should be more transparent not about the inside of the black box, but about what goes into the black box. To illustrate our claim, we iterated on the visual design and created the AI scorecard. This design shows how a fictive bank could explain how they use AI to make decisions for their products and services, and how they can be transparent more extensively.
AI scorecard suggests a new pattern for interfaces by adding a disclaimer saying which information is generated by an AI service. By clicking on this, the user can see a breakdown of all the information on the risks of AI misuse on that decision, the factors that are used in order of relevance, and an indicator of bias risk for each factor. This is ultimately accumulated in an overview page that shows the risk for all of the offered AI driven solutions accompanied by explanations.
While it’s possible to display the influence of data points in numerical values, bias cannot be communicated as such. It’s however possible to categorize bias as low, medium and high even though there are no strict rules for this. As stated above, the content of the AI is not human. But when the way of communicating AI with users is designed in relatable manners, it can be much easier for people to interpret the bias risks in the system.
To make AI fully transparent requires more effort from both people and organizations, as well as designers. We believe that the power to translate the data use of machine learning technologies into a relatable and interesting experience is how design can elevate the understanding of AI in society.
Next steps
As for the next steps, we aim to:
- Develop Transparent AI into a school-ready material and;
- Work on a unified communications guideline for organizations who harness AI to show them how the concept of communicating training and data influence applies to different types of AI.
Ultimately, we hope to change the perspective about AI as being a black box, and foster engaging learning and user experiences about data and how it’s used for machine learning systems in order to make transparency more concrete than merely a tool to criticize artificial intelligence.
