Deep Learning Interpretability for Medical Imaging: Regression Concept Vectors
Implementation and guidelines
In this post we present the implementation details of the iMIMIC at MICCAI 2018 paper Regression Concept Vectors for Bidirectional Explanations in Histopathology (paper preprint). The code runs in Python 2.7 or higher, with Keras and Tensorflow background (source code). GPU is preferred although the code works also on CPU-only systems.
Why interpretability?

Predictive performance is often the main objective when developing and training a deep learning model. In certain applications, however, the optimization of model accuracy might only be a partial solution of the problem and explanations are equally important. Moreover, as the complexity of the models is increased, model interpretability becomes difficult to achieve. This drawback in the optimization task has been addressed as formalization incompleteness [Doshi-Velez and Kim 2017]. In real world applications, we would like to be able to understand the reasons behind the model choices and we would like to know when the model is likely to fail and why. In the medical domain, for example, trust in machine learning methods is affected by the limited evidence about algorithm reasoning.
Feature attribution with Activation Maximization (AM) and Gradient-weighted Class Activation Mapping (Grad-CAM) heatmaps are very useful visualization methods for obtaining global and local explanations of Convolutional Neural Networks [Erhan D., et al., 2009; Selvaraju R.R., et al., 2017]. We recently applied these methods on patches of breast cancer histology slides (see Figure 1).

These approaches give interpretation only at the input pixel level. AM shows what combination of input pixel values would maximize class-specific activations, while Grad-CAM specifically highlights regions in the input that where the most important for the decision. Pixel values combinations, however, have a very low level of abstraction and it is difficult to map them to higher level human-concepts. Testing with Concept Activation Vectors [Kim B. et al., ICML 2018] introduced a tool to determine the importance of high-level concepts in classification tasks, such as the presence of stripes in the classification of a zebra.
But what if the concepts are continuous measures? Often medical diagnoses rely on continuous measures, such as tumor area, tumor diameter or radiomics features.
To give an example, the grading scheme of the Nottingham Grading System for breast cancer relies on measures of nuclei pleomorphism (such as nuclei size and shape). Tissue samples with larger nuclei sizes or enhanced contrast in the nuclei texture are generally assigned a higher score on the breast cancer grading scale.
The Regression Concept Vectors (RCVs) framework helps in detecting the importance of concept measures such as nuclei size or texture features in the network decisions [Graziani M., et al., to appear at iMIMIC, MICCAI 2018]. There are three essential steps to compute RCVs:
- Extract concept measures of interest
- Learn the direction of the RCVs in the activation space
- Determine the relevance of each concept measure
In the following we point to the implementation steps for the histology application as shown in the online github repository. More theoretical details can be found in the paper.
- Extract concept measures
Statistics about nuclei morphology are extracted from manual nuclei annotations and measures are averaged within patch.

# Nuclei morphology statistics
nuclei_morph = rcv_utils.nuclei_morphology(stats)
# Nuclei texture statistics
nuclei_text = rcv_utils.nuclei_texture(patches, nuclei)2. Learn the direction of the RCVs in the activation space
We solve regression in the activation space of a layer l. For instance, we regress the concept measure out of the activations for each input patch, as shown in Figure 2.

# inputs: vector of testing patches
# data_preprocessing removes the ImageNet mean from the images
inputs = cam_net.data_preprocessing(inputs)# we extract the network activations at layer l
get_layer_output = K.function([bc_model.layers[0].input],
[bc_model.get_layer(l).output])
feats = get_layer_output([inputs])[0]# we seek the regression line
reg_score, cv = rcv_utils. solve_regression(
X,
np.asarray(nuclei_morph[c])
)
3. Determine the relevance of each concept measure
Sensitivities scores are computed as the directional derivative of the network prediction on the direction of the RCV in the activation space (essentially, it measures how much an increase of a concept measure makes the network decision change.). Each sensitivity score gives a local explanation and refers to a single testing image.
# we compute the derivative of the network prediction wrt the
# activations in layer l for each input patch
derivative = rcv_utils.compute_sensitivity(
bc_model,
-1,
0,
np.expand_dims(test_inputs[p],
axis=0),
wrt_tensor=bc_model.get_layer('res4a').output
)
flattened_derivative = derivative.ravel()# we project the derivative on the RCV direction
sensitivity_score = np.dot(flattened_derivative, rcv)# for each test input we compute an individual sensitivity scoreall_sensitivities.append(sensitivity_score)
Global explanations can be computed by the bidirectional relevance score Br, which is defined as the ratio between the determination coefficient R² of the RCV regression and the coefficient of variation of the sensitivity scores (σ̂/µˆ).

# Br scores derive global explanations for all the testing points
Br = R * np.mean(all_sensitivities) / np.std(all_sensitivities)Conclusions
In this post we presented a step-by-step walk-through the implementation details for RCVs. When applied to a patch-based binary CNN (tumor vs non-tumor patches) for breast cancer histopatholgy, RCVs highlight the relevance of the concept measures nuclei texture correlation and contrast. Figure 3 shows that pixel correlation has a strong relevance in the classification of non-tumor patches, whereas contrast is relevant for the tumor class.

RCVs can be used during model development to get insights about network training. They could be extended to many other tasks and application domains and applied to different network architectures. Network retraining is not required, so their running time is relatively short () and they are also suited to CPU-only resources.

