Explainable Artificial Intelligence, or how to CONFIDE in a machine
Author: Iordan Iordanov, Chief Scientist at Corpy&Co., Inc.
Artificial Intelligence is virtually everywhere nowadays. Or it tries to be, at least. With the enormous potential of this seemingly incredible and powerful new idea, so many new applications have been proposed, and the possibilities seem to be limitless.
Only that AI is not a new idea. It has been around since the 1940s, in fact. The first scientific publication describing Turing-complete artificial neurons dates back to 1943. The idea of the perceptron, one of the most basic structures of a modern AI model, was first published in 1957. The only reason why there has been an explosion in AI research in recent years is that, at the time, we simply did not have the computational power to handle complex calculations on large amounts of data, and now we do. With computers becoming more and more powerful and relatively cheap, today everyone can train an AI model and try for themselves an object detection algorithm, for example. It takes less than 30 minutes, and you don’t even need to install anything on your own machine.
However, by the words of Ben Parker (Spiderman’s uncle), with great power comes great responsibility. Yes, we do have unprecedented computational power, and yes, we do have heaps upon heaps of data, and yes, everyone can train and run their own models today. The question is, should we?
As with all great ideas, one of the first thoughts that anyone can have about AI is, how can we make money with it? How can we sell it? Of course, the one who can sell it first will be the one to benefit the most. This is only one of the reasons why AI is (almost) omnipresent today — it is trendy, and any product based on AI is bound to receive a few aah’s and ooh’s. Quite important in a business setting, wouldn’t you agree?
As a consequence, we tend to get ahead of ourselves and dream of solutions for which we are not nearly ready yet, one of the most blatantly obvious examples being self-driving cars. Far be it from me to doubt tech geniuses claiming the availability of fully autonomous vehicles in 2020, but we have unfortunately seen in the past that self-driving cars are still not safe enough. Moreover, apart from technological limitations, there have been multiple cases in which reckless or improper use of the available technology has put human life in danger.
There is little we can do to prevent people from being… naive. We should, in fact, assume the worst-case scenarios when it comes to the human factor, and try to build resilient, intelligent, responsive systems that are able to account for any kind of mistreatment. Such systems should be able to deal with situations in which even a human agent would potentially fail to react properly. This is an ideal scenario, of course, but, again, the currently available technology is simply not ready to function even on par with humans. I insist on this point because it is the quintessence of this discussion, and because it has become the cause for unnecessary loss of human life.
Even earlier than the Uber accident in March 2018 in which Elaine Herzberg lost her life, there have been more than a few other cases in which people trusted their navigation systems too much. Such accidents have triggered a series of deep investigations into why AI systems fail to detect dangerous situations. In some cases, it turns out that the AI system has blind spots due to environmental conditions or due to the alignment of other objects; in the case of Elaine Herzberg, the system did see the woman, but decided that the detection is a false positive. In other words, the system thought that the detection was wrong so it did not take any action to stop the vehicle, leading to the loss of a human life. Very naturally, we ask the question: can we really entrust human life to an AI system that makes such mistakes? Just from these few examples, my answer would not be very favorable for AI systems. So, what can we do then?
First of all, we need to stop treating AI systems with reverence, and we need to stop fearing them. We might feel like AI is miraculous, and it is understandable that such incidents may generate distrust and hesitance, but we should keep in mind that AI in itself is not dangerous — it is its improper use that generates danger. We either fear or revere what we do not understand, so we should focus on understanding AI for two reasons: so that we can see it for what it is, and so that we know how to use it properly.
So, how do we go about understanding how an AI system works? You can ask any AI expert to explain what is a convolutional neural network, like the ones used to detect and identify objects. I am guessing that you would get an answer along the lines that it “extracts/learns/combines features” and based on these features it knows where objects are, and which object is what. When it comes to what a “feature” is and how it is extracted, the usual answer goes into the detection of lines, combined into more complex objects, combined into more abstract notions, and so on, and so forth. This is the general idea that everyone is based on, but the truth is that we still do not fully understand how a model learns to distinguish between cats and dogs, for example. The truth is that models guess, and the guess that seems more likely is the one that is chosen.
So, to be able to truly understand the logic behind an AI system, we need to understand how these guesses are made. For this purpose exactly, the idea of Explainable AI was proposed not so long ago. In 2016, the Defence Advanced Research Projects Agency (DARPA) of the USA opened a call for projects under the theme of XAI, aiming to promote the understanding of how an AI model works, and, by consequence, how it takes decisions. The aim was to deal with the so-called “black-box” property of AI systems, meaning that many models are opaque — we are supposed to feed them data, obtain a result, and trust that this result is correct without knowing how it has been obtained. If we consider for a second the fatal consequences of a mistaken choice by an AI system, as we mentioned before, the problems with this approach become obvious.
It should be noted that more often than not models are opaque unintentionally. Even if we had full access to the whole structure of a model, there are so many parameters involved that we would simply be overwhelmed by the sheer amount of considerations to take into account. How is anyone supposed to examine millions of parameters and make sure that not even one of them is amiss?
With these considerations in mind, more than a few methods have been proposed to get some insight into the black-box models that we use today. Some of these methods are based on visualizations of the attention of the model (showing the parts of an image that impact of a certain decision), others on neuron coverage (examining which neurons of the network are more active for specific known cases, and how many neurons are activated for different cases, globally), relevance propagation (seeing which features of a specific input are important for a decision), local approximations (used to capture the behavior of a model for a single sample by focusing on different parts on the sample), and so on.
We could talk about the technical points for a long time, but this is not our purpose here. What is important is to note that there is not a single method that is able to explain the decisions of any AI model, nor is there a common standard for what is a “good explanation”. What makes sense to an engineer might not be convincing to a lawmaker, or vice versa; yet, it is equally important to give a meaningful explanation to both. Explanations still need to deal with millions of parameters, and give an explanation that is simple enough to be manageable, yet complete enough to provide sufficient insight. Achieving this fragile balance is not a simple task. Users need to be able to interpret such explanations and take appropriate actions to eliminate potentially serious problems.
The topic of XAI is so important that even tech giants such as Google and IBM have launched their own services. However, even Google’s engineers are struggling with the difficulties of XAI. This is only indicative of the complexity and the importance of the subject. It is a very clear and unequivocal indication that instead of rushing into creating more complex systems, we ought to put the necessary effort into understanding the systems that we have right now. And it becomes rather obvious that it is crucial for those at the forefront of technological advancement to do so. In a very direct sense, it is up to us to make AI meaningful and to hold it accountable.
And this is exactly where my team and I make an actual contribution. I am the Chief Scientist at Corpy&Co., a tech startup in the heart of Tokyo. I have the privilege to work with many young and bright people from various backgrounds, and we all focus on the aspect of XAI to various extents. We are unique in what we do, because we are the only ones in our field who focus mainly on the explainability of AI models. We go even a step further, as far as working on Quality Assurance (QA) for AI models. Explainability is a tool for QA, since it can tell us if a model is reliable or not, and it can also reveal the conditions in which it is not.
As a team, we take it upon ourselves to push the boundaries of what we know. We ask the difficult questions and we dare to think outside the box, also from the point of view of a user. We have no delusions of grandeur or unrealistic goals — we know that we must advance step by step, steadily toward a firm grasp of what AI is, and how we can tame it. For this reason, we have created our own platform, CONFIDE. CONFIDE stands for all the values that we hold dear:
・Consistent — results are reproducible and reliable, no magic involved;
・Optimal — provide the best possible performance with the available resources;
・Normal — follow international quality standards whenever a standard is available, otherwise establish our own standards and diffuse them;
・Fair — promote fairness and equality with transparent AI systems;
・Interpretable — provide not only decisions, but also reasons and justifications;
・Dependable — a robust system that is able to account for unforeseen circumstances and handle them gracefully;
・Engineerable — always push the state of the art while ensuring that proposed solutions can be implemented in practice.
With CONFIDE, we are starting from concrete applications where AI is used, and we accumulate knowledge and expertise in understanding in-depth the inner mechanics of AI models in various contexts. Gradually, we are applying our knowledge to more mission-critical topics, including self-driving cars. In collaboration with academic and industrial leaders in the automotive industry, we are actively working on establishing safety protocols for AI systems, validating the performance of such systems, and making sure that these safety protocols follow the CONFIDE principle.
In this whole process, explainability plays a central role. More than the methods themselves, the concept that we need to know exactly what is happening inside an AI model, that we need to be able to explain it clearly and concisely to someone else, and that we need to be able to do that at any time, is the central philosophy that we follow.
In conclusion, AI is not the villain. It is just a powerful, yet extremely complicated tool, and we have not figured out how to use it properly just yet. Cheap computational power and publicly available resources make it easy for anyone to use this exciting new technology, but we need to be responsible and wield its power with full conscience of its implications. The fact that we can do something does not mean that we should do it. A good point to start is to try and understand how and why an AI system works, and a good starting point for this endeavor is to delve deeper into Explainable AI techniques. We owe it to ourselves and to society to make sure that the solutions that we propose are meaningful, safe, and transparent.
It’s up to us to save human lives and expand equality with AI.
That’s what we do at Corpy.
