Why computer says no? Explainable AI and other ways to collaborate with the black box

In our digital surroundings, more and more decisions are made by AI algorithms on our behalf now. It seems innocuous that the autoplay chooses for us what to watch next or facebook tries to guess what you would need from Amazon. When it comes to scenarios like deciding whether to kill the baby or the grandma for a self-driving car, spotting the outlaws from the crowd, telling how long you will stay in hospital or even alive, a mere ‘computer says no’ type of answer may not satisfy the devastated family member who just lost a loved one, an innocent person who ends up in jail, or the anxiety of facing your own mortality.

Issues about accountability arise as AI starts to get entangled with problems involved in social domains. An AI agent usually develops itself through learning from huge datasets, the hidden layers are not only opaque to a naive inspector but also become undecipherable to even its own developers. As a result, the algorithms become black boxes, of which only the inputs and outputs are known, the rest remains hidden. Hence, it is nearly impossible to understand why and how the algorithms generate the results. Considering the ethical and social impacts, it seems necessary to take action on increasing the transparency in AI’s decision-making process, so that computers would not only tell us ‘no’ but also why it is so.

“At the onset of this study our gut feeling is that modern tools of machine learning and computer vision will refute the validity of physiognomy (for inferring criminality), although the outcomes turn out otherwise.”

Researchers admitted in the paper that they are also surprised by the fact that machine learning algorithms are able to infer criminality from just face pictures.

The development of explainable AI, as the name suggests, is an attempt to open up the black box. By providing an explanation of the decision or prediction the AI has made, the human situated in the other side of the screen is able to gain more insights about the result as well as decide whether to trust the system. It is impossible to build a machine that provides explanations without first understanding how we humans give and receive explanations. One of the main insights from social science studies on explanation is that explanations are contrastive. A why-question often takes the form ‘why P rather than Q?’, in which P is the fact that requires explanation, and Q is a hypothetical case that was expected rather than P. Even when just asked about ‘why P?’, there is an implicit contrast case Q that helps to frame the possible answers and render them relevant. For us humans, we are able to infer the implicit contrast case Q based on context and common sense. However, for an AI model devoid of folk psychology and isolated from the rest of the environment, detecting something that is unsaid constitutes one of the challenges explainable AI is facing.¹

A generated contrastive explanation of two classes of the Iris dataset in a dialogue setting. The fact P is the ‘Setosa’ class, the contrast case Q is the ‘Versicolor’ class. In this case, the AI model is trained to classify Iris species, the contrast case Q is recognized with an additional locally trained AI model. Given this explainable AI model, you can extract a set of data points that define the data entry as either fact P or contrast case Q, after selecting Q, the points that define Q but not P are used to construct the contrastive explanation. The data point corresponding to the feature ‘pedal width’ is identified as playing the role that distinguishes ‘Setosa’ from ‘Versicolor’.²

Besides the challenge of selecting the most relevant contrast case to apply in a certain context, offering a satisfying explanation is also a social process. Comparing with a perfectly rational, logical computer agent, our human minds are terribly messy. We often explain the behaviour of others by the underlying intentions we attribute to them. An explanation like this usually consists of the beliefs, goals of the acting agent, and/or the emotions that trigger the action.³

An example of emotion-based explanation.³

Among the recurring insights from the studies about our reception of explanations are the general preference for simplicity and that there is no one-size-fits-all type of explanation. If providing contrastive and emotionally satisfying textual explanations are the only guidelines for developing explainable AI, the result may pass our Turing test on the task of communicating the AI agent’s action. However, the true understanding of how the black box actually works and where the results come from is indeed limited.

“… It’s like, how long are you gonna hug the shore with your boat, we gonna have to strike across the sea. And it means letting go of a lot of things, so we let go of WYSWYG and move to what I call WYSWYNC, what you see is what you never could, so that means that new visualisations of all kinds will be possible and none of them is outlawed, the point is what is useful.”

In an interview, Ted Nelson, one of the pioneers involved in the invention of hypertext, is addressing that the prevalent HCI paradigm WYSWYG (What You See is What You Get) renders the computer screens mere simulations of paper instead of encouraging explorations of new potentials for it to aid human thinking.

Not just the scientists who are dealing with AI or big data are concerned about the obscurity of black boxes. Now that social media platforms tailor and personalise content to individual users, our everyday experiences are increasingly mediated and shaped by the black boxes animating the platforms. Besides opening up the box and reading the coded instructions, one of the ways to get to know the working of algorithms is through speculative and exploratory experimentations.⁴ This method of trial and error has been a common heuristic to problem-solving in fields like cybernetics and computer science.

Similarly, by interacting and playing around with black boxes, we can see what input actions lead to what output results and start to make sense of how the hidden processes might work. When we are migrating to the digital space, the use of our body has somehow been neglected. The idea of embodiment emphasis that the meaningful experience is determined by the bodily engagement of a cognitive agent and the environment, from which knowledge about the world emerges. Without the involvement of the body in both sensing and acting, thoughts would be empty. According to the constructivist view of learning, from children on, we construct our knowledge about the world by actively playing in and exploring the environment.

At the TNO hackathon about explainable AI, the challenge was to explain the predictions of whether a diabetic patient will be readmitted to a hospital within 30 days, based on patients’ profile and medical data. Abdo Hassan, Ays Bilgin, Ahmed Mohamed, Emma Beauxis-Aussalet and myself have participated and are honoured to receive the first prize. Instead of providing a textual contrastive explanation, our approach provides an interactive data dashboard that allows the user to select different feature values, and see how the feature influences the prediction. In the above screenshot, the features with the solid large contour (in the _Patient_, _Diagnosis_, and _Medication_ boxes) show the AI prediction for a specific patient (top left box). The features with the dashed contour show a patient case for which the AI prediction would be different (top right box). The user can also explore different alternatives either by manually selecting different feature values or choosing suggestions generated by different explanation models, e.g. foil tree model from TNO², LIME, SHAP. All changes of the results from AI are communicated through direct visualisation. By tracking the user’s interaction history, this approach also opens up the possibility for in-depth user study and better personalisation.

Besides via the standard trio — keyboard, mouse, screen, our imagination of ways to collaborate with the black box should not be bound to just using eyes and fingers. Thanks to the advances in capabilities and miniaturisation of sensing technologies in recent years, the use of multisensory technology has been gaining more and more attention and interests, even for learning subjects that are abstract as arithmetic and geometry. In a natural environment, almost all events generate stimulations to multiple sensory modalities. Our brain has evolved to combine and integrate information from different sensory modalities to form a robust image of what is going on out there. This mechanism manifests itself in the form of illusion when the information from different sensory modality are in conflict. By the same token, our perception of a certain event can be enhanced by adding the congruent natural sensory signals. Therefore, engaging multiple senses can better approximate natural settings and produce greater and more efficient learning.⁵

In addition, the development of explainable AI is mainly about exploring alternative results at the moment. The truth is that even if we are able to see how the algorithm works, we never get the chance to understand a real ‘why’. For instance, in the above-mentioned case of explaining whether a diabetic patient will be readmitted to a hospital within 30 days, it shows that the chance of readmission decreases if we reduce the use of medicine. However, there is no way for us to understand why reducing the medicine makes the AI predict that the patient will not return to the hospital. Does reducing the medicine means reducing harm to the patient, or does it mean that the patient will be considered as not being sick at all? This is a limitation of data-driven AI in the first place. Data-driven AI is in essence processing statistics at a large scale, without reasoning or domain knowledge. Admittedly, it is a powerful tool to uncover hidden patterns underlying a huge dataset, when it comes to explaining the exact relationship between the features describing the input data points and the predicted outcome, most of the time it’s merely correlation instead of causation. To avoid possible mishaps resulting from an algorithmic decision, we need to be aware not to blindly trust the machine, but to conduct a closer inspection and apply our knowledge to interpret the results more humanely and, from designers’ perspective, consider how can we address or highlight such drawbacks via the interface.

A truly meaningful collaboration with an intelligent artificial agent is expected to aid us in comprehending complex situations, isolating the significant factors, and deriving solutions to problems. To achieve these objectives, not only shall we take advantage of the computer’s proficient mathematical skills, but also explore its capabilities for manipulating and displaying information that allows humans to apply our native sensory, mental and motor capabilities. In doing so, let’s hope that the machine will not be the replacement of human intelligence but as an augmentation to our cognitive capabilities.⁶

So hopefully next time when we ask computer the ‘why’ question, we can finally start to learn about what ‘42’ actually means.

References:

[1] Miller, T. (2017). Explanation in artificial intelligence: insights from the social sciences. arXiv preprint arXiv:1706.07269.

[2] van der Waa, J., Robeer, M., van Diggelen, J., Brinkhuis, M., & Neerincx, M. (2018). Contrastive explanations with local foil trees. arXiv preprint arXiv:1806.07470.

[3] Neerincx, M. A., van der Waa, J., Kaptein, F., & van Diggelen, J. (2018). Using perceptual and cognitive explanations for enhanced human-agent team performance. In International Conference on Engineering Psychology and Cognitive Ergonomics (pp. 204–214). Springer, Cham.

[4] Bucher, T. (2016). Neither black nor box: Ways of knowing algorithms. In Innovative methods in media and communication research (pp. 81–98). Palgrave Macmillan, Cham.

[5]Shams, L., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in cognitive sciences, 12(11), 411–417.

[6] Engelbart, D. C. (1962). Augmenting human intellect: a conceptual framework. PACKER, Randall and JORDAN, Ken. Multimedia. From Wagner to Virtual Reality. New York: WW Norton & Company, 64–90.

The Digital Society School is a growing community of learners, creators and designers who create meaningful impact on society and its global digital transformation. Check us out at digitalsocietyschool.org.