XAI: Explainable A.I. by Justification and Introspection

Gulfaraz Rahman
Jan 30, 2019 · 7 min read

Deep Learning is used to solve a wide range of problems by providing samples of input data along with the expected output from the neural networks. The learning is driven by concrete mathematical rules used systematically to adjust weights of the network — think of these as a collection of numerical knobs adjusted to produce the desired output. What the network actually learns are these numbers which transform an input question into an output answer. What do these numbers signify? Do they represent knowledge which can be interpreted and understood by humans?

Image for post
Image for post
Bombe — Wikimedia.

Let us begin at the two of the largest problem spaces where deep learning has proved itself — Vision and Language.

Using Neural Networks for Image Classification

Image classification is a task for which neural networks are used extensively. The performance of neural networks in this task is well established for many datasets [1]. The images in these datasets vary from images of digits to flora, fauna, people, cars, bikes, aeroplanes and other classes of objects. The process for image classification is to take an input image and pass it through a trained network where it goes through a series of transformations which gives a single output class.

Using Neural Networks for Text Generation

Another popular application of neural networks is in natural language. Neural networks excel at translation, language modelling, text classification, question answering, named entity recognition and dependency parsing. In late 2016, Google replaced its traditional NLP methods with an LSTM based method for its translation service [2]. This free service is available worldwide and is used by over 200 million people daily.

Image for post
Image for post
Text Generation using a Recurrent Neural Network.

The Need for Explainable A.I.

Although neural networks have been used to solve many problems, understanding of what the network actually learns is unknown. To give us an idea of the acquired knowledge, attempts have been made to visualize the learned weights. Classified as open research — we are yet to decipher the meanings of these tuned knobs. Without understanding what knowledge is actually acquired it is risky to use these networks in real problems.

The legendary tank detection network is a cautionary tale that warns researchers to overcome biases in the dataset. One version of the legend is quoted from [3] as follows,

…However, when a new set of photographs were used, the results were horrible. At first the team was puzzled. But after careful inspection of the first two sets of photographs, they discovered a very simple explanation. The photos with tanks in them were all taken on sunny days, and those without the tanks were taken on overcast days. The network had not learned to identify tank like images; instead, it had learned to identify photographs of sunny days and overcast days.

We cannot afford mistakes such as these in critical domains such as healthcare where real lives are at stake. Neural networks need to become more transparent or, at the very least, justify the predictions it makes.

Explainable A.I. can be viewed through various lenses. The approach depends on how we, as humans, are convinced. One may feel more confident about decisions if multiple models arrive at the same conclusion. Alternatively, the network could generate reasoning to support the prediction. Others may prefer the network grounds the prediction by highlighting parts of the input which helped make the decision — these could be words in an input sentence or regions in an input image. We will discuss two major types of XAI systems,

Justification Systems

Generating Visual Explanations [4] explores a model which produces sentences that explain why a predicted label is appropriate for a given image. The authors propose a ‘justification’ explanation system which produces sentences that explain why a certain classification decision was made. The Image Classification network is trained with an additional Text Generation network for the task of explaining predictions made by the former network. Thus when classifying unseen images, a sentence is generated to justify the class prediction for the given image.

Image for post
Image for post
Justifications generated by the model.

Introspection Systems

Grad-CAM [5] uses gradient information to produce a coarse localization map which highlights regions of the input image that led to the predicted class. The network's decision is introspectively traced from the last layer to the input image to visually explain which features of the input image contributed to the prediction.

Image for post
Image for post
GRAD-CAM highlights obtained by tracing gradients from the last layer to the input image.

We combine these complementary explanation systems to generate justifications which are supported by an introspective understanding of the network decision. This allows the model to generate less class-specific justifications and rely more on the features found in the input image.

The CUB Dataset

Why do we choose a collection of bird images? A fine-grained dataset of bird images [6] was used for this task because each bird can be assigned to a unique class. This property makes the Caltech UCSD Birds (CUB) dataset ideal for testing for class-specific general understanding and for image specific discriminative understanding.

How do these models fare in an adversarial setting? Can these systems be fooled by hackers? Congruent to other adversarial experiments our models fail when subject to such attacks.

Weakness to Adversarial Attacks

Adversarial attacks are when small perturbations are made to the input image/activation which is invisible to the human eye but leads the network to a wrong prediction. The Fast Gradient Sign Method, as described in [7], was used to generate the adversarial images. The caption generated by the explainable model for the perturbed bird image is completely undescriptive of the bird’s features. [8] is an excellent blog for a deeper explanation of adversarial attacks.

How can these explanation systems be used in the real world? We illustrate one application which constructs an explanation which informs why the given bird does not belong to a certain bird family.

Application: Generate Counter-factual Explanations

Given two distinct classes of bird images, a counter-factual explanation describes why one bird does not belong to the class of bird in the other image. These discriminatory explanations are useful for face matching, fingerprint searching, image-based location detection and similar tasks. spaCy’s dependency parser is used on the generated explanation and attributes are compared to discriminate the bird features found in the image pairs.

Image for post
Image for post
Bird images with explanations (Top Row). Counter-factual explanations — describe why the bird in the image does not belong to the class above (Bottom Row).

The discussed techniques are yet to solve the problems for which they were created. As mentioned earlier, the scientific community is actively looking for a deeper understanding of deep networks. The questions investigated are aimed to highlight that these open issues need to be addressed before we can achieve stable global adoption.

Explainable A.I. is essential to establish TRUST in Deep Learning solutions. Transparency is needed to guarantee fairness in the predictions made by models. Interpretability of the learned knowledge allows accountability of A.I. driven decisions. An ambitious motivation is a possibility of interpreting revelations when looking into these magical black-boxes.

Research conducted at the University of Amsterdam with Samarth Bhargav, Daniel Daza and Christina Winkler under the supervision of Prof. Zeynep Akata.


  1. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems 2012 (pp. 1097–1105).
  2. Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, Klingner J. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. 2016 Sep 26.
  3. Dhar V, Stein R. Intelligent decision support methods: the science of knowledge work.
  4. Hendricks LA, Akata Z, Rohrbach M, Donahue J, Schiele B, Darrell T. Generating visual explanations. In European Conference on Computer Vision 2016 Oct 8 (pp. 3–19). Springer, Cham.
  5. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV) 2017 Oct 22 (pp. 618–626). IEEE.
  6. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200–2011 Dataset. Technical Report CNS-TR-2011–001, California Institute of Technology (2011).
  7. Goodfellow IJ, Shlens J, Szegedy C. Explaining and harnessing adversarial examples (2014). arXiv preprint arXiv:1412.6572.
  8. Adversarial examples in deep learning — A Towards Data Science article by Gregory Chatel

Just A.I.

Simple and Reliable Solutions for Scalable Businesses

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store