How Visualizations Help Make Sense of Deep Learning
Deep learning poses many challenging questions. Visualization, through its exploratory and explanatory powers, can shed light on some of them:
- Why — why does the model work (interpretability)
- How — how well might the model work (on different subsets of data)
- What — what’s the performance of the model
We’ll look at a list of visualization tools, interfaces and techniques that can help with the exploration, interpretation or diagnosis.
Interpretability
Besides generic packages like SHAP and LIME, many interpretability tools are domain-specific as the models across domains could have very different mechanisms.
Vision
CNN explainer
CNN explainer is a standard-alone interface. It explains via interactive graphics how different convolutional layers work generically.
This package applies various image models explanation methods such as Grad CAM, occlusion sensitivity, smoothGrad, and integrated gradients, and can be applied either during training as a callback or on a loaded model.
These libraries provide feature visualization of CNN layers using deepdream style optimization. In activation atlas, the authors elaborated more on activation visualization and feature attribution.
NLP
RNNvis
RNNviz is a prototype tool to understand the hidden state of RNN by co-clustering the bipartite graph of hidden states and the word space.
BertViz
A python package offering the birds’ eyes view of attention across layers and headers, as well as query and key values of a particular attention head. The author Jesse Vig has a detailed writeup on how this works.
A python package by Jay Alamar with several interactive explanations of transformer models, such as ranking the output token across layers, or the one below which uses non-negative matrix factorization to visualize how groups of neurons spike in response to inputs.
With an example on Question-Answering system, this tool demonstrated its usage for scalable error analysis through a querying language and counter-factual inspections.
LiT (language interpretability tool)
This browser-based interface can be used to explore the behaviors of multiple NLP models, understand what kind of example does the NLP model performs poorly on, and if the prediction is attributed to adversarial behavior or due to training data priors. It has comparison mode for different models/examples. It is framework agnostic and supports Tensorflow/PyTorch etc.
How well might the model work?
This mainly could involve evaluating the model on different slices of data, individual datapoint, evaluate fairness, and adversarial robustness.
What-if tools
When one needs to slice-and-dice data, one could use What-if to facet by various features and inference scores. It integrates with various notebooks and TensorBoard. There’s some functionality to optimize for fairness. It offers counterfactuals for individual data points, but that doesn’t seem to apply to groups of data.
Performance monitoring
Tensorboard
Besides measuring performance metrics, having plugins such as the embedding projector, also has a debugger mode which helps identify numerical issues.
Source: https://www.tensorflow.org/tensorboard
The Embedding Projector
Ludwig’s Viz
Ludwig is an autoDL tool aiming to make deep learning more accessible by eliminating the boilerplate. Besides a large selection of supported models, it also has some unique visualization options. Those options include multiple ways of comparing classification models — such as when both models got the prediction right, both got the prediction wrong, and one model got it right, and other chart types which visualize multi-class probabilities of predictions for each class, or confidence threshold surface.
Source: https://ludwig-ai.github.io/ludwig-docs/user_guide/
HiPlot
Hiplot is an interactive parallel plot tool showing which combination of parameters renders the best outcome.
Source: https://github.com/facebookresearch/hiplot
Visualizing training data
As much of the model performance and robustness is dependent on quality data, having a clear view of the training data is also important. Here’s a tool to understand the characters of the datasets better.
Facets
One part of this tool is Facets dive, an interactive interface for visually exploring the data by attributes.
https://pair-code.github.io/facets/
Another part of it is Facet overview, which can be used through tf-data-validation package (tfdv) to visualize the distribution stats of the training set and to detect drifts between training and serving data.
We saw a variety of tools made to reveal model behaviors and data characteristics, address pain points by surfacing out-of-domain data, and facilitate actionable error analysis.
While these tools cover common use cases, other specific domains and models might call for more customized visual tools. And imaginably there would be unique challenges to create those tools given the nuance of ML.
When it comes to the generic qualities of great visualizations, Alberto Cairo listed 5 in the book Truthful Art. As I savor these qualities, I realize they can be re-interpreted in the context of ML:
- Truthful: showing result in a representative and consistent manner that does not mislead users to incorrect conclusions
- Functional: intuitive and flexible interface to explore or query potentially unstructured data like image or text
- Beautiful: easy-to-perceive encodings, sensible colors, and minimized cluttering/occlusion in lieu of massive data size
- Insightful: actionable insights on how to optimize the model or deeper understanding on its performance
- Enlightening: potentially revealing unknown unknowns
The list of tools mentioned is by no means a comprehensive list on the interplay between DL and Viz. But hopefully offers a glimpse of some major areas of focus 😎
Besides these tools focused on models, there’re also numerous visualizations made to illustrate specific concepts. More examples can be found at:
Distill
PAIR
UWashington Interactive Data Lab
Fast Forward Labs
IEEE VISxAI
