How Image Analysis and Natural Language Processing can be combined to improve Precision Medicine

In healthcare, there are a number of methods used to create different medical images. They include:

· X-rays,

· Computational tomography (CT) or computed axial tomography (CAT),

· Magnetic Resonance Imaging (MRI) and

· Modern nuclear imaging modalities such as PET and SPECT.

Scans such as CT, MRI and PET, have not only enabled the acquisition of much better images with higher resolution, they have also enabled the generation of many more images.

DICOM (Digital Imaging and Communications in Medicine) is the adopted format for the storing, exchanging, and transmitting of medical images which are typically stored in a picture archiving and communication system (PACS).

Image analysis is the extraction of meaningful information mainly from digitally stored images by means of by a number of processing techniques, each however may be useful for a small range of tasks as there still aren’t any known methods of image analysis that are generic enough for wide ranges of tasks, compared to the abilities of a human’s image analysing capabilities.

Natural language processing (NLP) on the other hand is a part of artificial intelligence where we apply computational techniques to the analysis and synthesis of natural language and speech. In the medical field, patient records usually contain a lot of important data that medical professionals need to extract. This information may include medications, immunization records and lab results.

However the majority of that information is free form text rather than structured into tables with rows and columns, meaning that processing the text in an automated manner and extracting meaning from it can be daunting, mainly due to the ambiguity of natural language.

However, there are NLP methods can be used to process, analyze, and interpret clinical text that allow medical professionals to ask important questions regarding an individual’s health.

NLP tasks can be separated into low-level tasks and high-level tasks. Some of these tasks have direct applications, while others are sub-tasks that are used to help solve larger tasks, with low level tasks usually feeding into high level tasks.

Examples of some low-level tasks include sentence boundary detection, tokenization, and problem-specific segmentations. While those for high-level tasks include spelling or grammatical error identification and recovery, word sense disambiguation, and information extraction.

Now, in the last 20 years there have been advances in both the field of image analysis and NLP.

The emergence of approaches such as multi-atlas segmentation techniques, fuzzy clustering, graph cuts, genetic algorithms, support vector machines, random forests or more recently deep learning — to name just a few have improved the analysis of images.

The application of deep learning and neural networks in natural language processing are just some of the advances that has brought about the creation of spoken dialogue systems and speech-to-speech translation engines, mining of social media for information about health or finance, and identifying sentiment and emotion toward products and services.

There have also been challenges with both fields. For instance, with image analysis demonstrating the need for more generic image analysis technologies that can be efficiently adapted for a specific clinical task. Also efficient approaches for ground truth generation are needed to match the increasing demands regarding validation and machine learning, along with algorithms for analyzing heterogeneous image data as well as to construct patient-specific models from medical images with a minimum of user interaction.

With NLP, there are three main challenges before it — language, context, and reasoning. With language, most natural language applications have yet to process it the same way human do, mainly due to the approach of treating text as data and not as a language as pointed out in an MIT Technology Review article, “AI’s Language Problem”.

The second challenge, is related to the first and has to deal with the understanding of context as any natural language text needs to proceed in the right context. This happens when the algorithms focus on the language structure and not just the words, as most current application currently do.

Then there’s the third challenge: the verification of the history and application of the reasoning that the natural language algorithms deploy to reach their conclusion.

There are some great opportunities that lie ahead if the challenges facing both image analysis and NLP are overcome, particularly when both are combined. For instance, the development of a diagnostic system that can not only efficiently read pathology reports by analysing images and the textual annotations that come the reports, but also generate a report of its based on analysis of the images using the clinical nomenclature and terminology in the same manner that a human would.

A situation where the image diagnostic system can be used is if for example, a patient goes to have a bone scan, the system can analyse the images from the scan and generate a report on the probability of how well treatment of cancer is working for cancer in the bone. By also analysing the text from other clinical inputs such as lab reports and clinical consultation notes, our diagnostic platform can also make precise recommendations on further treatment for the patient.

It will also be possible for the system to analyse images from a huge number of patients and be able to stratify the population based different clinical criteria, which can help policy decision makers formulate strategies based on the data extracted from medical image reports.

Acquisition of the data for image analysis and NLP in this manner have does bring up some ethical issues, as the data is personal for each of the patient. While there value in mining the data, especially for research purposes, people still have a right to decide on how and when their data is used. However the more data we have available for research, the more advances we can make not only in image analysis and NLP field, but in the improvement in the delivery of services around precision medicine and stratified healthcare.