Big Data, Medical Imaging and Artificial Intelligence
(Big Words folded into one project)
It is a rare occurrence when an engineer and an entrepreneur get to combine big data, medical imaging and AI into one project. But that is exactly what happened to me.
Let me introduce you to polyps. If you don’t know what polyps are then it probably means that you were not suspected of having some sort of growth.
This is a story of what I decided to do with these polyps, and how a bedroom discussion (more on that later) about polyps led me to develop a product. So, what are polyps you ask? Well, they are little creatures that live in your abdomen. They are there now as you are reading these words. In some cases, they grow to be malignant and dangerous to one’s health turning into what is commonly known as cancer.
It is a known fact that the leading reason of death in the world is caused by cancer. There are many forms of cancer, but Gastrointestinal Cancer, which grows from Gastrointestinal Polyps, is the most commonly occurring cancer of all.
At this point, I need to disclose that I am married to a physician. This blissful union has led me to conceive many ideas that later turned into products. Each day as my wife dons her surgical scrubs, she wages battle against these little creatures. From morning till the evening hours, she embarks on a quest to find them in order to prevent and hopefully cure her patients from the disease. When she returns home and we exchange details of our daily events, from how the kids are doing to what needs to be done around the house, we also turn our attention to ways of improving polyp detection in GI procedures (try it with your spouse and see how far you get).
Accuracy of diagnosis of any disease depends on a physician’s experience. Physicians are human, and humans make mistakes. In our case, physicians may miss polyps. Mis-detection of polyps may mean that malignant tumors could develop in a patient at a later stage.
Being a computer guy, and a serial entrepreneur I quickly concluded that computer assisted polyp detection systems can help reduce polyp mis-detection rates and assist physicians in finding the most important regions that need to be scrutinized.
This is the reason I developed a system called Sherlock — The Sleuth Able to Detect Gastrointestinal Polyps and Assist in Early GI Cancer Discovery.
My starting off point after completing the research phase and weighing the available options at my disposal, was to base my work on the assumption that only a deep learning model could make a difference for the scenarios I was trying to solve. I set off by collecting massive data sets of polyp images. Since I am blessed by the fact that I have a spouse owning an endless source of knowledge and information of trends in the field, It was easier for me to obtain relevant information regarding what a polyp is and what is not considered a polyp, and other clinical information required to validate my assumptions that would lead me on my path towards a viable product.
However, even with massive amounts of data at my disposal, I was not one-hundred percent clear on what I was looking for other than the basic need to find polyps. I learned that multiple forms of polyps exist — flat, adenoma, hyperplastic, and many more. For the sake of simplicity, I opted to look for any polyp, of any size and shape if it resembled a polyp. I also factored in clinical assumptions received from some of the best physicians I know in the field, however, these would go beyond the scope of my story.
can you spot the polyp?
Truth be told — a polyp is hard to spot. Especially when viewed in a video stream that produces images at 30 frames per second. Let’s not forget that a GI procedure is not static. The Endoscope is in constant movement, with varying rates of speed dependent on the physician who is operating the scope.
Enter “Big Data” into the mix. My next step was to train a deep learning model. How is that accomplished? By “feeding” the model with an endless number of images, for the model to accomplish its required task. But, to succeed in the training exercise, and that is the big “secret,” I needed to teach the model exactly what I was looking for, and where to find it in the image itself. The technical term for this step is called “Object Classification.” It is achieved by annotating the image. Annotation was done manually. Why? Because if this was an automatic process that would mean that a product already existed!
I ended up studying 1,000 polyps. In order to feed the deep learning mechanism, I had to process at least 10 video frames for each polyp. As no procedure resembles the other, I had to contend with various degrees of lighting, Endoscope or organ position and many other variables. I processed over 10,000 images that needed to be annotated. Pardon the pun, but do you get the picture? In the field, annotations are performed by physicians, from medical students learning their craft to the most experienced of physicians. No small feat.
The next step was the development of an annotation tool to mark the polyp. As this whole project was about how to go about discovering polyps, annotations had to be extremely accurate. This is an example of a marked polyp mask:
While conducting a brainstorming session with deep learning experts, I came across an interesting article describing the “Hotdog not Hotdog” application and project developed for HBO’s “Silicon Valley” program. The project started as a joke, when someone invented an application that captured an image of either a “hotdog or not a hotdog.” The person capturing the image needed to share it (send it to a server) and mark it as a “hotdog” image or not. This project became the basis for a deep learning model that became public for images of hotdog detection. I found countless articles describing the phenomena. All of them came to the same conclusion. When you train your model with a negative background of your expected prediction, it improves total prediction accuracy.
The next step was to create background Frames. Background Frames were made of the same location and scenario of captured polyps, but without the polyp itself. I virtually took the polyp out of the image using various methods, from simply relating to re-positioning of the frame, to others based on image processing techniques using a Laplacian Pyramid extraction method of the subject.
I started to “feed” the images shown above to the deep learning model with two labels (“object classification”): “Polyp” — marking the region of the polyp as shown in the image; and, “No Polyp” for the entire scene.
The entire process of data collection, annotating data and preparing the pre-process training algorithm took over eighteen months! During this time, I observed how the classifier (the name of a mature deep learning model), grew to become an extremely powerful tool delivering consistently accurate predictions.
I continued testing the model with new images and procedures received from various sources. This allowed the system to be re-trained and re-tested on a continuous basis and more importantly providing increased confidence in the predicted results. In the end, I was able to conclude my work by providing what is called in the medical field a “Confusion Matrix.” This matrix provides a rough description of the testing results of a deep learning model, explaining the success and failure points with the sensitivity and accuracy of the model.
As glamorous as this may sound, at the end-of-the-day producing a nice algorithm served me and me alone (and those nightly pillow conversations with my wife). I had to struggle with the fact that what I achieved in laboratory conditions needed to take place in real-time during a procedure with a patient on the table. I needed to be sure that I could achieve automatic detection of polyps. Like with most physicians I spoke with, my primary sounding board (my wife) was extremely skeptical and challenged me to take this entire process one step further. They claimed that this detector was good in theory, but there was already one fictional detective. Sherlock Holmes. Unknowingly my wife named my next product. Sherlock.
The challenge was to turn this entire experiment into a viable product.
One of the primary lessons learned was that the post-processing calculation for the algorithm was using up a tremendous amount of CPU power, meaning that a lot of calculations were based on statistics derived from Convolutional Neural Network (In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery). I had to consider that I was working with video content and each frame had to be detected and examined during a procedure, as close to real-time as possible. My primary challenge was to reduce CPU overloading.
After significant experimentation I concluded that if I was able to lower the probability and confidence levels, I would be able to produce a predicted region for mapping at a much faster rate. Lowering the confidence levels to 80% meant that if the algorithm suspected an 80% probability that a specific region was a polyp, it should be marked and shown. In talking to the medical experts at my disposal I learned that in field conditions (i.e. medical examinations) this was more than enough.
Further work led me to add another feature that became a staple of the product. I named it the “beep effect.” As detected regions of polyps were presented in the video and marked as white or red rectangles, as the scope is manipulated during the examinations (and often very quickly), these rectangles blink on and off very quickly. So, if a polyp was discovered by Sherlock and its on-screen duration was less than one second, a physician would find it hard to notice. So, I added a “beep”. An alert to focus a physician’s attention on a specific region.
Developing this feature was a project in and of itself. The timing of the beep needed to be fine-tuned. There was no use in using the feature each time a polyp probability presented itself. I feared that this would create a “crying wolf” effect. The system would be over protective. Additionally, there was no use “beeping” for polyps previously discovered. Identifying the polyps and leveling the primality for the beep was another project (A primality test is an algorithm for determining whether an input number is prime. Among other fields of mathematics, it is used for cryptography. Unlike integer factorization, primality tests do not generally give prime factors, only stating whether the input number is prime or not). One interesting outcome from developing the “beep effect” was that a physician upon hearing the beep, intuitively slowed his examination of the patient giving a more thorough scan of the region. This vastly improved the rate of detection.
The result of this work is a viable product called “Sherlock” in honor of those late-night pillow talks with my wife. The system is still learning, detecting and improving. For me, it was a great way to engage my passion for big data, deep learning (virtual reality in a glance) and medical imaging.
And just in case you were wondering if my wife and I stopped our late-night pillow-talks? Guess again.
Did you know that your tongue is a story teller? More on that later.
