What is XAI — Explainable AI and Visualization (Part 10)
This article is a continuation of the research for the study ‘Explainable Deep Learning and Visual Interpretability.’
The interest in interpretability and transparency has led to an emerging field of research called Explainable Artificial Intelligence (XAI). XAI is a concept that explains decisions made by machine learning models and provides justification in a way interpretable by humans. It enables an understanding of overall strengths and weaknesses and conveys an understanding of how the system will behave when implemented in the future and how to correct the system’s mistakes.
The XAI research and proposals offer a shift towards more transparent and explainable AI system. It aims to create a suite of techniques that produce more interpretable models while maintaining high-performance levels. This area of research holds substantial promise for improving trust and transparency of AI-based systems.
The Defense Advanced Research Projects Agency (DARPA), a division of the Defense Department of the United States that explores innovative technologies, has launched a large initiative in XAI, funding several projects that aim to make artificial intelligence more interpretable and explainable. The approaches range from adding further machine-learning systems geared toward providing an explanation to the development of new machine-learning techniques where the elucidation is embedded by design. The initiative involves collaboration with research institutes and academic partners to identify challenges in problem areas and develop interpretable models and explainable AI systems.
Historically, there has been occasional interest in the explanations of intelligent systems, but the recent successes and proliferation of AI for large-scale applications and the use of increasingly complex and non-transparent algorithms necessitate another rise of interest in the need to better understand these systems.
The recent wave of interest in explainable artificial intelligence can be attributed to the latest efforts by the DARPA’s initiative to fund XAI projects. Defense intelligence agents and military operations may come to rely heavily on machine learning to parse huge quantities of data from satellite imagery, radio signals and other sensors and to control the growing equipment of autonomous systems.
The US military wants to take pre-emptive measures to ensure that this reliance doesn’t lead to blindly trusting in any type of algorithm. It might not matter that much if the new machine learning technique is opaque, but not in the case of a military commander trying to identify a potential target or an intelligence officer surveilling cyber espionage by a foreign agent.
DARPA program managers recognize that there is a real advancement happening in AI and that these models are very opaque and difficult for people to interpret, especially if they are not an expert in AI. Thereupon, they are committed to helping develop AI techniques that provide an explanation for their reasoning.
DARPA is funding several research groups which are pursuing a range of approaches and techniques to make AI models more explainable. Recently, it released Explainable Artificial Intelligence (DAR, 2016) to encourage research on this topic. The primary goal of XAI is to create a suite of machine learning techniques that produce explainable models to enable users to understand, trust, and manage the emerging generation of Artificial Intelligence systems.
Furthermore, the military will be developing several autonomous systems in the coming years that will inevitably rely on machine learning techniques like self-supervised learning and reinforcement learning. Self-driving military vehicles, along with aerial drones, will be deployed, and they will become increasingly capable over time.
XAI has been gaining increasing attention lately, with several conference series dedicated exclusively to the topic of interpretability, transparency, fairness and accountability. Broadly, the XAI landscape can be divided into two research groups (i) a group of academic institutions, non-profit research labs and firms in the regulated industries (such as credit, insurance and legal system) and (ii) civilian and military researchers funded by DARPA. The former group is mainly focused on promoting fairness and transparency in algorithmic decision-making with social and commercial impact.
Deep Learning Visualization Research
Deep learning visualization research is an interdisciplinary area spanning multiple related fields within and outside the deep learning and AI community. In academia, the primary venue for deep learning visualization research consists of two main groups (1) Information Visualization and Visual Analytics community; and (2) artificial intelligence and deep learning communities. These two groups occasionally also coordinate with human-computer interaction communities, which in turn is a multi-disciplinary field of computer science, cognitive science, human-factor engineering, design and others.
In this section, I discuss the research questions focused on neural network interpretability that formed the basis of the research process and discuss them using the human-centered framework, which is organized around various facets of deep learning visualization research. Finally, I also discuss the visualization toolkit and open-source libraries available to visualize and interpret deep neural networks.
Why would one want to use visualization in deep learning?
One of the most important reasons why one would like to visualize deep learning is to understand how deep learning models make decisions and what representations they have learned so we can better trust the model and expect the model to behave in a certain way in the real world.
Interpretability & Explainability
Though the idea of model understanding mainly concerns the black box problem of the machine learning models, it is a neural network in particular that encounters the issue of black box models because of their opaque nature and complex working mechanism.
Visualization is a powerful tool to deconstruct the learning process of the neural network and explore the network’s automatically-learned features. It’s also useful to gain a fine-grained perspective of the network architecture and qualitatively evaluate the model’s predictions. Currently, interpretation and explanation often are the most common motivations for deep learning visualization.
Deep learning models are harder to interpret than most machine learning models due to their inherent complexity and nonlinear structure. The underlying decision-making processes for why the models are achieving such performance are challenging and sometimes mystifying to interpret. Therefore, in practice, users often approach them as a black box and cannot explain how learning from input data was done nor how performance can be consistently optimized, due to which the development of the model largely depends on the trial-and-error process. It could also be detrimental when the model does not perform satisfactorily; users are not able to understand the causes or know how to fix the problem.
An example application of model exploration is deployed on Facebook’s machine-learning platform. ACTIVIS is an interactive visualization tool for interpreting large-scale deep-learning models. It is integrated with multiple coordinated views, including a computation graph that provides an overview of the model architecture and a neuron activation view for pattern discovery and comparison.
Debugging & Improving Models
Building a deep learning model is an iterative design process, and finding the right combination of parameters and settings for a particular network depends on multiple factors. The appropriate selection of parameters can vary widely even when performing very similar tasks due to various reasons.
These parameters include network structure, depth and width of hidden layers, error bound, optimization algorithm, finely tuned hyper-parameters such as learning and regularization rate and the data vector used are often selected in a trial-and-error process. Therefore, when designing neural networks, it is tough for both beginners to select the right parameters and find the correct general rules. While, even for deep learning experts, it largely remains a trial-and-error process.
Model development also constantly requires monitoring of the model during the training phase, identifying misclassified instances and testing a subset of data instances to observe performance and study sample results. These critical steps help guide model developers by suggesting them possible directions to explore and validate or invalidate their initial assumptions. Thus, a visual analytics tool can help model developers build and debug their models while accelerating the iterative development process.
Comparing & Selecting Models
A visualization toolkit can be an essential tool for model developers and users to compare various different models and select the right one for the problem at hand. Selecting the best-performing model requires inspecting the model metrics to compare network architecture, accuracy, loss, and hyperparameters used, and visualizing parts of the model to evaluate and pick the one that has the highest accuracy, generalize well with the real-world data and avoid pitfalls such as overfitting.
Experiential Learning
Visualization can be used to teach deep learning concepts and build an intuition for how neural networks operate. It is useful to educate beginners and non-expert users about AI and spark curiosity and learning among the non-technical audience. An ideal example of an experiential learning application that uses interactive techniques to engage beginners and non-experts in the learning process is TensorFlow Playground.
TensorFlow Playground is a web-based interactive visualization of neural networks for education and intuitive learning and understanding, where users can play with a basic neural network by changing different configurations such as the width and depth of hidden layers, managing dataset ratio and batch size of the dataset, activation type, learning and regularization rate, and types of non-linear activation units. It utilizes two-dimensional toy data sets for problem types such as classification and regression tasks. The manner in which each node in the network is activated across different input data values is fully visualized as a heatmap in a two-dimensional space.
What data, features and relationships can be visualized?
The technical component of the neural network that can be visualized is the network architecture, learned model parameters (edge weights and convolutional filters), activation units (activation output), high-dimensional space and model metrics, just to name a few.
When is visualization useful in the deep learning process?
Visualizing deep learning is useful during both the training and inference processes. During training, it can be used to visually explore and rigorously observe the model as it learns and track the model performance. The real-time update is useful for observing the metrics recomputed after every batch and epoch, such as loss, accuracy and training time. These metrics aid model developers with many insights.
Visualization after training has been used to discern feature visualization and attributions to understand what patterns are learned by the model filters and what features are attributed to the model decisions. For the prototype development discussed in the methodologies section, I use a pre-trained model to visualize the activation outputs of a convolutional network to show what part of the input instance attributed to the classification decision.
Who would use deep learning visualization?
It can potentially help three broad groups of practitioners and users of AI who stand to benefit from deep learning visualization and visual analytics: model developers, model users, and non-specialists.
The next article in this series covers the specific research methods and hypothesis for the XAI research experiment.
