Picture perfect: AI + medical imaging
— November 2017 —
Introduction: AI’s potential in radiology
Artificial intelligence (AI) in medical imaging is a hot topic. It’s also a controversial one — questions regarding both its ultimate potential and clinicians’ future roles have drawn considerable attention.
Notably, in late 2106, Google’s Geoffrey Hinton — the AI pioneer now heading up the Vector Institute for Artificial Intelligence — proclaimed that
Hinton has since acknowledged the considerable overgeneralization of his original statement. In a subsequent conversation with Siddhartha Mukherjee of The New Yorker, he clarified —
“The role of radiologists will evolve from doing perceptual things that could probably be done by a highly trained pigeon to doing far more cognitive things…
As reported by Mukherjee, Hinton ultimately constrained AI’s likely nearer-term participation in radiology to reading X-rays, CT scans, and MRIs. Further, he hypothesized that, at some point, “learning algorithms will make pathological diagnoses.” Artificial intelligence might facilitate a clinician’s activities, but without necessarily assuming control. According to Hinton,
“Early and accurate diagnosis is not a trivial problem. We can do better. Why not let machines help us?”
The buzz around AI has created a narrative of machines potentially rendering humans unnecessary. The reality, however, is that even relatively simple tasks performed routinely by radiologists can’t yet be delegated to AI — at least not without a high level of human supervision.
Flavors of AI
The gap between narrow AI & artificial general intelligence (AGI) cannot be overstated. AI in radiology is narrow AI and doesn’t approximate AGI in any way.
At the moment, humans provide detailed instructions to power AI. Both the task and the methods to complete the task are supplied to the computer running AI software. This is narrow AI and is the driver behind such technology as Google’s AlphaGo and IBM’s Watson. A combination of training data and processing algorithms are used to address a unique task. While remarkable, the applicability of each instance of narrow AI is critically constrained.
A distant cousin to narrow AI, AGI refers to intelligence “…that could successfully perform any intellectual task that a human being can.” Key questions, such as whether consciousness is requisite for general intelligence, remain unanswered. At the very least, the abilities of AGI would not be constrained to a single task, and would include 1) capacity to gain knowledge and 2) ability to adapt the ways in which knowledge is applied. Given the myriad complex tasks performed by radiologists, some form of AGI would be required to perform the role of a human radiologist to any substantive degree. Such AGI is simply nonexistent, so unlikely to be overtaking the radiologist’s job anytime soon. As noted by Dr. Paul Chang from the University of Chicago in a RSNA 2016 interview with Aunt Minnie, radiologists have adapted to new technologies in the past and are continuously reinventing themselves — a skill that’s certainly not in the narrow AI toolbox.
concerns about general AI
How does AI work (CNN, deep learning — basicly section III) apriori work from the 90’ , changes that lead to the renessaince of AI
Why has the coupling of medical imaging and radiology come into the spotlight recently? Historically, medical imaging has conducted decades of research and development on applying artificial intelligence techniques to challenges in image processing, segmentation, and analysis. References of a 1992 review article on neural networks in medical imaging include papers discussing supervised and unsupervised learning, self-organising neural pattern recognition machine (1987), self-learning neural networks in electrocardiography (1990), and training of ECG signals in neural network pattern recognition (1990).
To answer this question, consider the recent explosion of effort on all things AI — broadly speaking, not only in medical imaging — which is almost entirely a result of advances in GPU technology which facilitate low-cost, widespread access to computational power sufficient to execute artificial intelligence techniques. AI techniques which were available decades ago, but were impossible to implement in practice, are enjoying a revival. In medical imaging, this advance in GPU technology made it possible to evolve beyond earlier computer-aided detection (CAD) algorithms to deploy AI techniques which employ computationally expensive algorithms.
The reason that computational expense is the key driver is related to the success of a specific AI approach known as deep learning. Specifically, deep learning with convolutional neural networks (CNNs).
Deep learning systems based on CNNs ultimately identify patterns in new data after having been ‘trained’ to identify patterns of interest in annotated data. Convolution is, roughly, a method of weighting input data to achieve a desired output. In the case of CNNs, the convolutions are performed on neural nets — weighted maps which associate multiple inputs to outputs, and several layers of maps may be used to increase performance.
Why the computational expense? Deep learning with CNNs involves iterative fine tuning of the neural net weightings via training on a very high number of well-characterized, detailed datasets. Until recently, computation times for deep learning with CNNs were entirely impractical. The advent of rapid-processing GPUs has solved that issue, enabling a new focus on — and progress with — deep learning techniques which have proven highly successful for many applications, including radiology.
The key difference between an AI approach and a CAD approach to medical imaging? In CAD, humans provided instructions on what to look for. In AI, humans provide instructions on how to look. The AI software is provided with examples of the element of interest and given a computational map for evaluating ‘success’. The AI is trained to associate individual examples with their specific labels or annotations (outcomes). Armed with a sufficiently large training set, the AI algorithms guide the process of characterizing the element in comparison to previously characterized (by a human) instances of other images/data. Not to be confused with a 1:1 correspondence yielding a yay or nay result, a more suitable way to think of the comparison process is to consider that the computer has memorized thousands of patterns corresponding to, for example, the labels ‘lesion’ or ‘no lesion’. By assessing the likelihood of whether a new image includes matches to these patterns (or combination of patterns), the AI generates a decision on whether or not the image includes lesions. If so, boundaries can be produced from this decision data, labels generated, and quantification can proceed based on the code/instructions provided. (The quantification process is more similar to traditional CAD than to AI.)
What can AI do now in the daily routine of a radiologist (clinical studies and uses that show it’s capabilities (section IV) , big players IBM,GE, future role of AI
AI is expected to to reduce the time radiologists spend on repetitive, low-level tasks. For example, Lee et al. predict it will deliver “quantitative analysis of suspicious lesions, and may also enable a shorter time for reading due to automatic report generation and voice recognition”. Such quantitative analysis might extend to organ volume, specific angles, ratios, etc.
Otherwise obscured information may be detectable with AI. It can also serve as a second set of eyes, and provide opportunities for experts to extend the reach of their accumulated wisdom.
— ? Sandor’s note: “This list could be followed by the natural language processing compressed and moved below (mark I.)” —
This last point is reflected in a recent Wall Street Journal article which reports that IBM is working toward deep learning systems intended to multiply each expert’s influence.
“The goal is to scale the expertise of the clinician,” says John Smith, an IBM research fellow and manager of multimedia and vision at IBM Thomas J. Watson Research Center. “The computer can see a lot more data than any clinician can ever see.”
Charles Koontz, chief digital officer of GE Healthcare, shares a similar perspective in the same WSJ piece —
“What we’re developing is a suite of applications to make them [people] more effective.”
Concerns regarding AI overextending its own influence to ‘take over’ are poorly supported. The fact remains that generalized AI is not yet here and we’re still well in the realm of narrow AI. Kai-Fu Lee, a former Google China CEO, discussed narrow vs general AI in Wired in July 2017:
Given a massive amount of data in a particular domain, deep learning can be used to optimize single objective functions, such as “win Go,” “minimize default rate,” or “maximize speech recognition accuracy.”
These AIs run “narrow” applications that master a single domain each time, but remain strictly under human control. The necessary ingredient of dystopia is “General AI” — AI that by itself learns common sense reasoning, creativity, and planning, and that has self-awareness, feelings, and desires. … But General AI isn’t here. There are simply no known engineering algorithms for it. And I don’t expect to see them any time soon. The “singularity” hypothesis extrapolates exponential growth from the recent boom, but ignores the fact that continued exponential growth requires scientific breakthroughs that are unlikely to be solved for a hundred years, if ever.
Can I guarantee that scientists in the future will never make the breakthroughs that will lead to the kind of general-intelligence computer capabilities that might truly threaten us? Not absolutely. But I think that the real danger is not that such a scenario will happen, but that we won’t embrace the option to double down on humanity while also using AI to improve our lives.
The state of the art in the realm of medical imaging supports the idea that we are still some ways off from automated systems which work seamlessly and intelligently to take over human roles in the clinic. As Mukherjee reports in the New Yorker:
Some of the most ambitious versions of diagnostic machine-learning algorithms seek to integrate natural-language processing (permitting them to read a patient’s medical records) and an encyclopedic knowledge of medical conditions gleaned from textbooks, journals, and medical databases. Both I.B.M.’s Watson Health, headquartered in Cambridge, Massachusetts, and DeepMind, in London, hope to create such comprehensive systems. I watched some of these systems operate in pilot demonstrations, but many of their features, especially the deep-learning components, are still in development.
AI: deep learning and convolutional neural nets
Deep learning systems in medical imaging applications almost exclusively utilize convolutional neural networks (CNNs). The editors of IEEE Transactions on Medical Imaging’s 2016 special edition on deep learning in medical imaging note that CNNs require large amounts of labeled data and that the availability of large, appropriately annotated medical data sets is hampered by privacy concerns and proprietary use. Furthermore, they note that rare conditions are, by their very nature, underrepresented in the data. Techniques such as transfer learning pose potential solutions, but sufficient data is still required to fine tune from generalized learning on large scale, annotated natural datasets (such as ImageNet) to specific learning on datasets including the actual features of interest. Aiming to provide guidance on the quantity of training data required, Researchers at Massachusetts General Hospital and Harvard Medical School developed a method for evaluating performance as a function of number of training sets; in practice, however, researchers determine the minimum required training cycles via trial and error.
Which brings us to the question of available data for training cycles, which is a particular struggle in medical imaging as specialized equipment and environment are required to generate the images. Also, privacy is an important consideration for any form of patient data and appropriately labeled training data is ill suited for posting in a public repository without extensive preparation of the data to ensure it is both anonymized and appropriately annotated. Kohli et al. explore the fact that “everyone participating in medical image evaluation with machine learning is data starved” in a 2016 whitepaper. Of benefit to deep learning researchers, Andrew Beam of Harvard maintains a GitHub repository with a curated list of Medical Data for Machine Learning with links to data. While access to sufficient and appropriate data continues to be a challenge, resources such as ImageNet and Beam’s repository are first steps toward a solution.
Challenges in / Barriers and limits of the development of AI (see below)
In regard to the goal of automating processes, it is worth noting that there is no single, definitive deep learning architecture or corresponding parameter set for medical imaging generally, nor even for specific imaging applications. Moreover, new advances, such as implementation of fully convolutional networks (FCNs), will further crowd the playing field. It’s not clear that there will ever be a complete eradication of the human component in deep learning systems.
Currently, deep learning systems for medical imaging require considerable human input to function as intended, even in cases where such input is being minimized. As an example, consider the previously mentioned automated muscle segmentation study. The researchers demonstrated promising results for automated muscle segmentation on CT images for body morphometric analysis, with segmentation time reduced from 30 minutes to 0.17 seconds and the introduction of an average error of less than 3.68%. The Massachusetts General Hospital and Universitaetsmedizin Berlin researchers concluded that “The fully automated segmentation system can be embedded into the clinical environment to accelerate the quantification of muscle to provide advanced morphometric data on existing CT volumes and possible expanded to volume analysis of 3D datasets.” Of note, however, the methods section details the researchers’ tweaks to standard FCN algorithms (variations on the fusion of layers) prior to deployment, as well as their empirical determination of both the optimal learning rate and weight decay parameters.
The promise of automated systems streaming relevant, reliable data to clinicians is an exciting one, as is the promise of automated — and improved — imaging. The ultimate potential for deep learning in medical imaging is as yet unknown.
The May 2016 IEEE Transactions on Medical Imaging special edition on deep learning noted that neural networks were applied to lung nodule detection as early as 1993, and in May, 2017, Erik Ridley of Aunt Minnie reported that “The use of AI in radiology actually isn’t new; thousands of papers on computer-aided detection (CAD) and image analysis algorithms have been published in journals and presented at meetings over the past 30 years.”
Such CAD systems, however, tended not to be “learning systems” with dynamically and iteratively adaptive computational algorithms aimed at achieving recognition without relying on deterministic models; rather, the standard approach was to optimize image data fit to such a priori models. The distinction is important to grasp as medical imaging CAD efforts in recent decades have yielded numerous disappointing results, and almost none have been widely adopted.
Deep learning techniques applied to medical imaging will help move the needle toward the goal of truly intelligent imaging; further computational advances by researchers on the bleeding edge, such as Hinton, will accelerate the process. In anticipation of this, startups in the space are plentiful. A recent issue of Diagnostic Imaging highlighted three machine learning startups focused on medical applications and, along similar lines, nanalyze published a two-part series on startups focused on AI for medical imaging. For example, Arterys uses its cloud-based platform to bring “deep learning to medical imaging, starting with cardiac analysis”, with a segmentation feature currently available, and features in development including prediction, tracking, classification, and detection. While cloud-based solutions pose special challenges for data privacy, Arterys’ technology ensures that “patient data stays within the hospital and is accessible by physicians from anywhere”. It remains to be seen whether the level of security such an approach provides proves sufficient.
The major players in radiology are deeply invested in their own efforts. For example, GE coordinates research efforts with several academic and clinical research collaborators. GE’s partnership with UCSF aims to develop “deep learning algorithms aimed at delivering information to clinicians faster” while a GE partnership with PartnersHealth (Massachusetts General and Brigham & Women’s) is “pinpointing what’s been holding AI back and developing the business model, platform and tools to ensure clinicians and patients can benefit from its potential.” IBM’s Watson needs no introduction and has its toes in the healthcare pool, as well. According to nanalyze,
“While best known for playing Jeopardy, Watson has applications across many industries including healthcare. “Watson Health Imaging” is a key part of Watson Health, and something which has been built over the years with some very strategic acquisitions of medical data.”
IBM has a team developing its Cognitive Radiology Assistant, a system which “…analyzes medical images and then combines this insight with information from the patient’s medical records to offer clinicians and radiologists support for decision-making.” Ultimately, it is still very early days for deep learning in radiology and there is actually little in the way of deep learning powered offerings for widespread clinical use from the major medical imaging technology providers. Yet.
Countless clinical studies exist in which researchers test various deep learning implementations on real data. In Annual Revue of Biomedical Engineering’s 2017 June issue, Shen et al. “introduce the fundamentals of deep learning methods; review their successes (in) image registration, anatomical/cell structures detection, tissue segmentation, computer-aided disease diagnosis or prognosis” and also summarize a variety of clinical results.
Recent studies associated with specific clinical imaging applications include:
Brain
Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions, Akkus et al, 2017.06
3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study, Dolz et al, 2017.04
Predicting Deletion of Chromosomal Arms 1p/19q in Low-Grade Gliomas from MR Images Using Machine Intelligence, Akkus et al, 2017.08
Breast
Unsupervised Deep Learning Applied to Breast Density Segmentation and Mammographic Risk Scoring, Kallenberg et al, 2016.05
Detecting and classifying lesions in mammograms with Deep Learning, Dezso Ribli et al, 2017.09
Endotracheal tube position
Deep Convolutional Neural Networks for Endotracheal Tube Position and X-ray Image Classification: Challenges and Opportunities, Lakhani, 2017.08
Kidney
Performance of an Artificial Multi-observer Deep Neural Network for Fully Automated Segmentation of Polycystic Kidneys, Kline et al., 2017
Liver
Computerized Prediction of Radiological Observations Based on Quantitative Feature Analysis: Initial Experience in Liver Lesions, Banerjee et al, 2017.08
Lung
Central focused convolutional neural networks: Developing a data-driven model for lung nodule segmentation, Wang et al., 2017
Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks, Setio et al, 2016.05
Towards automatic pulmonary nodule management in lung cancer screening with deep learning, Ciompi et al., 2017
Multiple sclerosis
Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation, Brosch et al, 2016.05
Muscle, morphometric analysis (cancer)
Pixel-Level Deep Segmentation: Artificial Intelligence Quantifies Muscle on Computed Tomography for Body Morphometric Analysis, Lee et al., 2017
Orthopedic
Artificial intelligence for analyzing orthopedic trauma radiographs: Deep learning algorithms — are they on par with humans for diagnosing fractures?, Olczak et al., 2017.07
Stroke
Artificial intelligence in healthcare: past, present and future, Jiang et al., 2017.06
Thyroid
Thyroid Nodule Classification in Ultrasound Images by Fine-Tuning Deep Convolutional Neural Network, Lee et al, 2017.08
Whole body MRI
Fully automatic, multiorgan segmentation in normal whole body magnetic resonance imaging (MRI), using classification forests (CFs), convolutional neural networks (CNNs), and a multi-atlas (MA) approach, Lavdas et al, 2017.07
Summary (section VII)
It is impossible to consider the future of deep learning in radiology without acknowledging the fact that it will need to earn its place in a highly regulated environment; with deep learning’s ‘black box’ nature, one other obstacle to clinical implementation will be satisfying the FDA of a new device or clinical software platform’s safety and efficacy. Although we have yet to see how that will play out, new (or risky) medical devices have previously been successful in gaining FDA approval, and the groundwork is being laid; the first FDA approval for a clinical tool employing machine learning was awarded at the beginning of 2017 for Arterys’ Cardio DL.
As noted in Stanford’s 100 Year study on artificial intelligence, “For decades, the vision of an AI-powered clinician’s assistant has been a near cliché” and the goal of deploying AI to support — not replace — radiologists seems to be guiding the way forward. Armed with this perspective on the role of artificial intelligence in medical imaging, we might assess the radiologist’s evolving role somewhat more positively than might be expected in view of Geoffrey Hinton’s (original) remarks on the matter.
It remains to be seen how, exactly, artificial intelligence will permeate not only medical imaging, but also clinical workflow and coordination of disparate sources of data. In the highly complex and highly regulated world of medical devices, AI is a tool at the disposal of engineers and clinicians — and its tremendous potential to contribute to the continual improvement of medicine in its ultimate goal of effectively and efficiently diagnosing and treating patients is starting to be realized.
In addition to the time freed up for cognitively intense work once AI kicks in to perform the mundane tasks still required of many radiologists, potentially exciting developments include a tremendous volume of new data-knowledge (previously out of reach due to computational expense), the extended reach of expert opinion via contributions to ubiquitously employed training sets, and mitigation of clinical errors via the first fundamental theorem of probability — the law of large numbers. With appropriate human oversight, AI in radiology has a good shot at remaining on the right side of Do No Harm… but do we even know what ‘appropriate human oversight’ looks like? That may be the ultimate challenge, and not just for radiology.
END
