Trust in AI: Explainability Perspective
Artificial Intelligence (AI) is a human-like behavior displayed by a machine or system trying to achieve a meaningful output, and repeatedly improving itself with incoming information. We can think of it as a system that mimics human behavior using an extensive amount of data and is highly complex for humans to understand. AI models are often considered to be black-box models which we don’t fully understand how they work and even though black-box models often outperform humans and model performances are evaluated by their predictive power in general, it does not mean that a high-accuracy model is always the most valuable solution on its own.
Transparency is one of the key elements to achieve trust in AI
Only 50% of adults around the world say they trust companies that use AI as much as they trust companies that don’t (Ipsos Survey WEF, 2022). To enhance trust in AI solutions, the end-user must be able to understand what AI is, why AI chooses one output over the other, and they must know that AI will increase their well-being and won’t cause any harm. The long-term validity of AI depends on society’s trust, and, at that point, transparency is crucial for establishing and maintaining society’s trust. Society should be confident that AI systems which have a wide range of high-stakes usage, such as healthcare, loan disbursements, human resources, legal decisions etc. do not cause any unintentional harm, and that we should accept transparency as a high-priority concern for high-stakes decisions. Without clarity in the decision phase of AI, using black box AI becomes challenging for businesses, which could potentially be the biggest obstacle to gain mass adoption as it drives the lack of trust.
The transparency term defined in European Commission Ethics Guidelines for AI as closely linked with the principle of explicability which include traceability, explainability and communication, and encompasses transparency of elements relevant to an AI system: the data, the system, and the business models (AI HLEG, 2019).
So, how do we create transparency in AI? Traceability, explainability and communication, which are the main elements of transparency, can be achieved with “Explainable AI (XAI)” which is a perspective designing AI models to make them human-understandable. We can assume there are two main ways to achieve explainability in AI: one is to explain black box AI models and the second is to develop high-accuracy interpretable white box models (Figure 1). XAI can provide an understanding of which features affect outputs, with what features in the data a model is used in learning, and thus, if the model is biased or inaccurate.
Convincing society about AI
According to the 22nd Annual CEO Survey of PWC, the vast majority (84%) of CEO’s agreed, “AI-based decisions need to be explainable in order to be trusted”. This is more than CEO’s percentage who believe that AI is good for society (79%).
So why do companies need explanations to be able to trust AI? There are lots of examples in domains like health, finance, fraud etc. that we can mention. Suppose you developed a high-accuracy black box model to predict whether a person is fraudulent or not. After model validation, you decide to use the model that has high prediction performance and the related business satisfied on the accuracy level. But, at that point, the business unit hesitates to use it because they don’t know the reasons for AI decisions. In practice, when an end-user who is affected by an AI decision, flagged as a fraudulent, requests an explanation, the business unit will need to make an explanation to the end user and, if they are asked to, they will also have to give the same explanation to the regulators as a legal requirement. Here, making a rational explanation is crucial for a company’s reputation as well as complying with legal requirements. Instead of saying “we don’t know, the machine just did it”, being able to convince the end user and the regulators about decisions produced by the machine provide the trust we need to enhance the validity and acceptance of AI.
Explainability comes with many benefits…
We can discover business insights and relationships that we did not notice even before. Especially for use-cases that AI performs better than humans, features that affect the target may be far different from what we think or know before. At this point, we can come to a reverse engineering point that a high-accuracy explainable model provides us a knowledge which we have missed as humans. Imagine an AI algorithm discovered the relationship between a disease and a factor that we haven’t discovered until now. Wouldn’t that be an amazing discovery?
As well as gaining new unexplored insights, we can also prevent wrong decisions made by AI, by detecting the features that are actually unrelated, but AI model finds related, or that the underlying data is not sustainable. Hence, we can reduce mistakes and thus the cost of mistakes, especially for decisions that have a significant impact on humans. By detecting biases and drifts with explainable AI, expert opinion will also be able to contribute to the model. This way, modeling will be in line with the business perspective, the model performance will be maximized by understanding potential weaknesses, and resultingly the AI model will be more applicable and reliable for both the stakeholder and the end user.
Being able to interpret AI models, ensuring their compatibility with expert opinion, detecting and ability to modify incorrect predictions will provide us a greater understanding on the model. Such an enhanced explanation ability of AI models completes a considerably crucial part for accountability for regulative parties. Explainable AI models can manage regulatory compliance, reduce risk, minimize manual operations and of course the risk of it, namely, decrease unconscious harm from AI.
In some cases, AI systems can produce unfair and biased outputs. The Amazon use-case is one of the most popular examples. Amazon decided to shut down its AI algorithm which is accused of systematically downgrading women’s CVs for some technical jobs. The company could not find the real reason of bias and any way to overcome the gender-discrimination issue. There have been many different opinions on whether the algorithm really makes a sexist distinction or not and an explanation that everyone agreed on was never made. If we look from a different perspective, what if the algorithm were interpretable? The company would most likely know the reason for the unfair gender-based decisions, easily explain the decision, and easily make the necessary modifications to remove the bias, perhaps even notice unfair labeling before deployment, and we would never have talked about the unfairness of the algorithm. Being able to interpret the model provides us to see if there can be any unfairness in the model or when an unfairness is detected, so that we can easily understand the reason of unfairness or if it is really unfair or not.
In conclusion,
The popularity of XAI is gaining momentum as an emerging methodology that provides understanding about AI decisions. With the increase of technologies, amount of data, the use of technology, the need for explainability to adapt to these new developments of AI is gaining importance for companies. Outputs produced by AI are not pieces of a virtual scenario; they are actually used as answers to real life questions and have real consequences. People are content with AI when it doesn’t affect a crucial side of their lives, say, when they are asking for a shopping recommendation. But for high-stakes decisions, the game is changing, dissatisfaction is taking place with AI decisions. At the end of the day, we need a trust strategy because many companies need the insights of machine learning and the need to adopt AI is now more important than ever.
