Anyone can publish on Medium per our Policies, but we don’t fact-check every story. For more info about the coronavirus, see cdc.gov.

Don’t Just Scan This: Deep Learning Techniques for MRI

Nicholas Bien
Feb 7, 2018 · 9 min read

Written by Nicholas Bien

Deep learning will soon help radiologists make faster and more accurate diagnoses. (image source)

Over the last decade, the ability of computer programs to extract information from images has increased tremendously. We owe most of this advancement to convolutional neural networks (CNNs), a type of neural network specialized for processing image data. CNNs have consistently outperformed classical machine learning (ML) techniques (e.g. support vector machines, random forest, k-nearest neighbors) since 2012, when AlexNet won the ImageNet Large Scale Visual Recognition Competition (Krizhevsky et al. 2012). Most of the work of designing a classical ML algorithm lies in choosing appropriate features. In contrast, a deep neural network takes raw input (possibly after some preprocessing) and automatically learns features through training. It is thus essential to understand what data and architectures are best for the task at hand.

With that in mind, this article attempts to:

  1. provide the necessary medical background and preprocessing tips for deep learning research on MRI
  2. introduce CNN architectures for segmentation
  3. survey current state-of-the-art MRI segmentation methods
  4. discuss remaining challenges to improving deep learning models for MRI

MRI Basics

At a high level, MRI works by measuring the radio waves emitting by atoms subjected to a magnetic field. The appearance of tissue in an MRI depends on the tissue’s chemical composition and which particular MR “sequence” is employed. The most common of sequence is T2-weighted MRI, in which tissues with more water or fat appear brighter due to their relatively high number of hydrogen atoms. In contrast, bone (as well as air) has low signal and appears dark on T2-weighted images. For brain MRIs, T1-weighted, T1-weighted with gadolinium contrast enhancement (T1-Gd) and Fluid Attenuated Inversion Recovery (FLAIR) are commonly used sequences along with T2-weighted images (Isin et al. 2016). In prostate cancer diagnosis, a different combination of sequences called multi-parametric MRI (mpMRI) is used (Sarkar et al. 2016). Determining which sequences to use for a given disorder or body part requires careful research or radiological expertise.

Brain MRIs labeled by sequence type. The far right image is a radiologist‘s segmentation. (Havaei et al. 2016)

The deep learning task

MRI preprocessing

  • Registration. If the patient moves during an MR screening, images may be offset from one another. If different sequences are combined in a single channel, or if a 3D network is used, then the images must first be aligned to a common orientation. In datasets hosted for public challenges, this is typically done before the data is released.
  • Bias field correction. MRI images are affected by bias field distortion, which causes the intensity to vary even across the same tissue (Pereira et al. 2016). The N4ITK method (Tustison 2011) is the most common method for correcting this.
  • Normalization. Due to the nature of MRI, even images of the same patient on the same scanner at different can have different intensities. Many MRI segmentation models use an intensity normalization from Nyul et al. (2000) to alleviate this problem. Additionally, as is typical with CNNs, each input channel (i.e. sequence) is normalized to have zero mean and unit variance within the training set.

CNNs for Segmentation

The goal of semantic segmentation is to predict a class for each pixel. The architecture above shows the “patch-wise” approach (Ciresan et al. 2012).

In 2014, Long et al. (2014) introduced fully convolutional networks (FCNs) for semantic segmentation. This network utilizes an encoder-decoder structure, where input image passed first through layers of convolution then through layers of upsampling to produce pixel predictions of the same size as the original input. SegNet (Badrinarayanan et al. 2015) improves upon the original FCN, and achieves state-of-the-art performance today (Garcia-Garcia et al. 2017). Since the encoder-decoder architecture generates all pixel predictions at once, it is faster than patch-wise segmentation. An additional advantage is that a pre-trained CNN for classification can be used for the encoder portion of the network (however, the lack of large MRI datasets makes this less useful for MRI segmentation).

Example of encoder-decoder segmentation by SegNet (Badrinarayanan et al. 2015)

Other segmentation architectures attempt to explicitly solve the problem of local dependencies while still using a patch-wise network. For example, in a cascading architecture, one CNN outputs class probabilities for each pixel, which are then fed to a second CNN for the final segmentation. Another approach is to apply conditional random fields or other probabilistic models during post-processing.

The principles of 2D segmentation carry over well to 3D segmentation, but with an increase in computational complexity. Because of this, 3D volumes are sometimes segmented slice-by-slice by a 2D CNN then reconstructed. V-Net (Milletari et al. 2016) is a good example of a fully convolutional 3D segmentation network, which happens to be for segmenting prostate MRIs. The medical imaging field would particularly benefit from techniques for making 3D segmentation algorithms more efficient.

State-of-the-art MRI Segmentation

where P represents the segmented area and T represents the ground truth area. Dice scores range from 0 to 1, where a score of 1 represents perfect segmentation. Each model in the BRATS challenge receives three Dice scores, one for each part of the tumor (whole, core, and active).

Isin et al. (2016) summarizes the best models on the BRATS 2013 dataset to date. We analyze the two best automatic (as opposed to semi-automatic) models, as well as the best 3D model. The models, along with their dice scores, are as follows (for reference, a human rater scored 0.88, 0.93, 0.74):

In all of the models, the last dimension of the input is for sequences rather than colors. All applied some preprocessing, with Pereira et al. being the most extensive. All three models also made use of a post-processing technique whereby connected components below a certain size threshold are removed from the prediction. Pereira et al. augmented the data using 90º rotations, while Havaei et al. and Urban et al. did not find data augmentation helpful.

Havaei et al. chose not to pursue a 3-dimensional network because the spacing between between MRI slices wasn’t consistent. However, Urban et al. found comparable results with a relatively simple 3D patch-wise network. Pereira et al. introduced smaller filter sizes in the convolutional layers in order train a deeper (11-layer) network. Their network also adds dropout to the fully connected layers to reduce overfitting. The architecture of Havaei et al. is the most complex, with two cascading CNNs, one for generating class probabilities and one for generating final predictions. In dealing with unbalanced class sizes — roughly 98% of the pixels are healthy — Havaei et al. first train with an equal number of healthy and unhealthy patches, then re-train only the output layer with a more representative distribution. (A more common way to handle this is to weight misclassification of unhealthy pixels more than misclassification of healthy pixels.) How the strengths of these and other models can be combined remains to be explored.

Remaining Challenges

  1. Data isn’t readily available in large quantities. Most online datasets for medical imaging contain hundreds of images, whereas ImageNet has 14 million. More collaboration between healthcare and tech is needed to establish a better pipeline for data acquisition.
  2. What is ground truth? Radiologists often disagree significantly on the segmentation or diagnosis called for by an MRI. Deep learning models can often deal with random variability in ground truth labels, but any systemic bias in radiology will persist in deep learning models trained on radiologists’ predictions. On the flip side, a deep learning system that exceeds human performance could inform new discoveries in radiology, just as medical imaging problems are stimulating new developments in deep learning today.

I am extremely grateful to Matthew Lungren MD MPH, Assistant Professor of Radiology at the Stanford University Medical Center, and Bhavik Patel, MD, MBA, Assistant Professor of Radiology at the Stanford University Medical Center, for their guidance and valuable feedback. I would also like to thank Pranav Rajpurkar, Jeremy Irvin, Shubhang Desai, and Tanay Kothari of the Stanford Machine Learning Group for their comments.

Further Reading

Brain MRI

  • Isin et al. 2016. Review of MRI-based Brain Tumor Image Segmentation Using Deep Learning Methods. (review w/ BRATS 2013 scores)
  • Chen et al. 2017. VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images. (3D, residual learning a la ResNet)
  • Brosch et al. 2016. Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation. (3D, multiple sclerosis)
  • Dou et al. 2016. Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks. (cascade architecture)
  • Milletari et al. 2017. Hough-CNN: Deep learning for segmentation of deep brain regions in MRI and ultrasound. (voting system, 2/3/2.5D)
  • Kleesiak et al. 2016. Deep MRI brain extraction: A 3D convolutional neural network for skull stripping. (3D, skull stripping)
  • Kamnitsas et al. 2017. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. (conditional random fields)

Prostate MRI

  • Milletari et al. 2016. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.
  • Liu et al. 2017. Prostate Cancer Diagnosis using Deep Learning with 3D Multiparametric MRI.
  • Tian et al. 2018. PSNet: prostate segmentation on MRI based on a convolutional neural network.
  • Litjens et al. 2016. Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge.

Cardiac MRI

  • Tran 2016. A Fully Convolutional Neural Network for Cardiac Segmentation in Short-Axis MRI.

Breast MRI

  • Moeskops et al. 2016. Deep Learning for Multi-task Medical Image Segmentation in Multiple Modalities.

Knee MRI

Stanford AI for Healthcare

Artificial Intelligence to Improve People’s Lives.

Thanks to Tanay Kothari, Pranav Rajpurkar, and Jeremy Irvin

Nicholas Bien

Written by

Stanford AI for Healthcare

Artificial Intelligence to Improve People’s Lives. Learn more at https://stanfordmlgroup.github.io/

Nicholas Bien

Written by

Stanford AI for Healthcare

Artificial Intelligence to Improve People’s Lives. Learn more at https://stanfordmlgroup.github.io/

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store