Research Methods — Explainable AI and Visualization (Part 11)
This article is a continuation of the research and design for the topic ‘Explainable Deep Learning and Visual Interpretability.’
Method Selection
The methodologies section covers methods chosen to investigate the research problem. I begin this section by restating the research problem and underlying assumptions underpinning this study: interpreting and understanding deep neural networks using interactive visualization to synthesize concepts of deep learning to non-specialists, inspire human curiosity and learning, as well as broaden people’s access to interactive tools for deep learning.
Scientific Methods
The primary method selected for the research process is the scientific method. I selected this method as it involves careful observation and empirical study. Science involves the empirical study of nature, and the scientific method has been the hallmark of several leading research in this interdisciplinary field. Indeed, much like a child’s brain evolves to become a model builder and learn to learn, researchers in these fields, over time, examined and pondered how to learn from nature by employing the scientific method in their research process.
Deep learning is primarily a method for machines to learn from historical data and classify patterns that are loosely modeled on the way a biological brain learns to solve problems. This empirical method, coupled with the detailed mathematical representation of the model and their numeric computation, makes the scientific method an ideal choice of method for this research.
In this research, I use tools of the scientific method for both performing experiments and raising application-agnostic questions of the inner working of the fundamental building blocks of the deep learning models, which ought to be examined with the tools of the scientific method to make sure we not only comprehend the effect but also begin to understand the cause which is the raison d’ etre of science.
Although the scientific method is widely discussed and written about, it remains pure in its core concept: observe a phenomenon of interest, formulate a mathematical model, verify how the model accounts for past observations, and use it to predict variants of the phenomenon and future observations. And following Popper, models shall be readily abandoned if falsified by observed data. However, falsifiability is often largely insufficient: a specific dataset may concomitantly support models based on distinct and perhaps incompatible assumptions. It is thus necessary to identify extensions to existing measurements and examine where conflicting models may deviate appreciably from each other and thereby provide grounds for additional testing and falsification.
The scientific method consists of two stages: (i) formulating hypotheses and the (ii) testing them. What differentiates this from other forms of methods is the second stage: subjecting hypotheses to empirical testing by ascertaining whether or not predictions derived from hypotheses are borne out in relevant observations and experiments. Hypotheses and assumptions are the initial stages of scientific inquiry because they incentivize seeking truth and a hint of where to find it.
The underlying principles of the scientific method are essential for evaluating the hypothesis, enhancing perspective, increasing productivity, and inspiring innovation. These principles include logic, probability, parsimony and hypothesis testing, as well as science’s presuppositions, limitations, principles and bold claims of rationality and truth. Beyond such methodology, some practical issues are shared broadly across the sciences, such as implementing effective science education and relating the scientific enterprise to the humanities.
Additionally, I also employed other research methods to support our research processes, such as action as research, agile development and system design. The selection of a research methodology was dependent on the research question itself and how best it can be addressed: interpreting deep neural networks.
Research Hypothesis
While deep neural nets learn efficient representations and deliver superior performance, most users consider them as a ‘black box’ system due to their opaque nature and complex working mechanism. As in most cases, they learn representation and patterns that is difficult to extract and present in a human-readable form. While this is true for certain types of deep learning models, it is not entirely true for an image recognition model because they are representations of visual constructs and abstracts which can be deconstructed to see what shapes and patterns were detected by the model at successive layers and how does that correlate to the prediction class at the output.
Typically a neural network is provided with an instance, for example, an image or audio segment, and it computes transformation on this instance at successive layers until it finally produces a probability value for a prediction. Inside the network, each neuron at each layer detects a specific feature through training and multiple epochs. The information flowing through the neural network is a set of numerical weights, parameters and connections that provides no direct hint as to how the task is performed or what the correlation is between inputs and output data. Further, the parameter selection is random and adjusted iteratively throughout the training process.
My analysis of previous work suggests that deeper representation in a CNN capture high-level abstracts or visual concepts. More importantly, convolutional layers naturally retain the spatial information of the input data, which is otherwise lost in the fully connected layers. Thus I expect the last convolutional layer to retain better features, which is balanced between spatial information and class semantics.
The neurons in the last convolutional layer mostly observe the class-specific information in the image, such as object parts. Therefore, I use gradient descent to compute the gradient of the activation output of the last convolutional layer in CNN to understand the importance of each neuron for a class of interest. We use relevant heat map visualization to highlight the importance of each neuron (channel or filter) and its correlation to the prediction made by the model.
Building upon this assumption, we formally set up our hypothesis as follows: Apply image localization and object detection techniques to distinguish what part of an input image attributed most to the classification decision by the model. In other words, localization is a useful method to study which pixels in a given input have high activations or to what input neurons are most receptive and how that correlates to the class prediction.
Finally, with the growing complexity of the deep learning model, the critical need for understanding its inner working has increased. I postulate interactive visualization is a potentially powerful technique to fill such a critical need. It can be used to deconstruct the problem of a black-box system and able to inspect what features the network has learned, patterns discovered by filters and uncover other hidden structures in the data.
The next article in this series covers the design and prototyping of the XAI research experiment.
