The Advent of Black Box Models — Explainable AI Visualization (Part 7)
This article continues the background research for the study ‘Explainable Deep Learning and Visual Interpretability.’
Deep learning models are harder to interpret than most existing machine learning models. Due to its many layers and parameters, multiple types of non-linear activation functions and randomized gradient descent training process, understanding these models remains a fundamental challenge. Because of this, the deep neural network is often considered a “black box” Although there has been some effort later in the machine learning and human-computer interaction community to shed light on their inner workings, most observers would acknowledge that neural networks as a whole remain a black box.
Generally, the information stored in a neural network is a set of numerical weights and connections that provide no direct evidence of how the task is executed or the association between inputs and outputs. The opaque nature of these models also limits their usage and acceptance in high-end science and engineering applications since it is demanded to use methods and techniques based on functions that can be understood and validated.
Another reason that complicates the use of the neural network is that there are no well-defined criteria for choosing a neural network structure and corresponding parameter selection. It mostly depends on the trial and error process.
The problem stems from the fact that we cannot merely inspect the deep neural network to see how it works or how it performs parallel computations. A network’s learning and reasoning are embedded in the behavior of thousands of simulated neurons, arranged in dozens or even hundreds of intricately interconnected layers.
For example, in the CNN model, neurons in the first layer receive an input image (as RGB value), which performs computation and outputs a new signal. These signals are fed into the successive layer until the final output is generated. Additionally, a process known as backpropagation adjusts the calculation of neurons to optimize the output. The backpropagation algorithm is the standard training method that uses gradient descent optimization to update the parameters.
Further, traditional analysis with a dense network is complicated because of the complex array of high-dimensional data involved. Advanced algorithms discern many varied patterns in the dataset, which they leverage to make decisions. The user has an idea about what went in but has no idea what patterns were recognized or were most significant to arrive at the solution given out.
The next article in this series covers the societal implications of the black box models.
