This work : https://gandissect.csail.mit.edu/ came out recently. From their abstract:
In this work, we present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level. We first identify a group of interpretable units that are closely related to object concepts with a segmentation-based network dissection method. Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output. Finally, we examine the contextual relationship between these units and their surrounding by inserting the discovered object concepts into new images. We show several practical applications enabled by our framework, from comparing internal representations across different layers, models, and datasets, to improving GANs by locating and removing artifact-causing units, to interactively manipulating objects in the scene.
The demo is incredibly fun and compelling — I highly recommend playing around with it. In it, you can “add” trees to a generated GAN image by applying a brush. What the brush does is increase the activations of internal nodes that have been found (through a separate process) to be correlated to generating images of trees. There are some surprising constraints from this work, the main one being that you can’t just put trees anywhere in the image — the image context only licenses trees in certain locations.
The other thing is that it points at how little we understand about how these deep nets operate. It’s interesting that there is a specific node (or a set of nodes) in the deep network that licenses tree generation — why wouldn’t they be generated from a more distributed representation?
And ironically, perhaps we now are finding a place where traditional neuroscience has something to add to our understanding of deep networks. The same ways that neuroscientists probe how the brain functions (segmenting vision for example into multiple stages) with ablation studies, stimulation, and FMRI perhaps map onto learning about deep networks. And maybe even psychologists have a role here in assessing exactly what the deep network has learned.
(Thanks to David Rosenberg for conversations and insights in the piece)