Visualizing Pathologies in Ultrasound Image using OpenCV and Streamlit
Problem Statement
We have participated in Detecting Pathologies Through Computer Vision in Ultrasound Omdena challenge to build an Ultrasound solution that can detect the type and location of different pathologies. The solution works with 2D images and also can process a video stream.
Identify the presence of a specific pathology on the ultrasound image and provide the location of the pathology with bounding box coordinates and mask. Ultrasound is a relatively inexpensive and portable modality of diagnosis of life-threatening diseases and for use in point of care. This will assist to deliver impactful and feasible medical solutions to countries where there are significant resource challenges.
Introduction
Visualizations help highlight the location of the tumor but also give us the intensity of the tumor at each point in AOI (Area of Interest).
The app takes the image and mask generated from a model as input and gives us the bounded box, mask outline, heat map.
Streamlit
Streamlit is an open-source framework for creating interactive apps in a short time entirely in python. The main advantage is its compatibility with other libraries like Matplotlib, OpenCV, NumPy, pandas, and many more. Installation is very easy!
pip install streamlitOpenCV
OpenCV is a Computer Vision library. We use it in our app to get the contours, draw them on images, and generate heatmaps using color maps. We can install OpenCV using pip.
pip install opencv-python-headlessLet’s define the steps needed to build this app with code
The APP CODE
We first import the dependencies.
import streamlit as st
from PIL import Image
import matplotlib.pyplot as plt
import pandas as pd
import cv2
import numpy as np
from matplotlib.patches import RectangleRead Data
We define a function to read images as inputs. We use file_uploader from streamlit to accept only png, jpg, jpeg file formats for images and Pillow’s Image to open image files.
def read_image(name):
image = st.file_uploader("Upload an "+ name, type=["png", "jpg", "jpeg"])
if image:
im = Image.open(image)
im.filename = image.name
return imShow Visualizations
The show image function takes an image, mask, and finds the contours in the mask i.e white object boundary from the black background. CHAIN_APPROX_NONE stores all the boundary points. RETR_EXTERNAL returns only extreme outer flags. All child contours are left behind.
Then, the image and contours are sent to _bbAndMask and _heatmap functions to get a bounded box, mask outline, and heatmap.
def show_image(image, mask):
mask = cv2.cvtColor(np.array(mask), cv2.COLOR_BGR2GRAY)
cnts,_= cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
_bbAndMask(image, cnts)
_heatmap(image, cnts)The bounded box and mask outline are shown in a single figure with two axes.
We use Matplotlib for generating the visualizations and st.pyplot shows the visualizations in the Streamlit app.
def _bbAndMask(image, cnts):
fig, (ax1, ax2) = plt.subplots(1, 2)
ax1.axis('off')
ax2.axis('off')
_bbox(image, cnts, ax1)
_maskOutline(image, cnts, ax2)
st.pyplot(fig)Bounded Box
We get the bounded box coordinates using OpenCV's boundingRect and add Rectangle as a patch to the axis.
def _bbox(image, cnts, ax):
ax.imshow(image)
for c in cnts:
area = cv2.contourArea(c)
if area < 10:
continue
[x, y, w, h] = cv2.boundingRect(c)
ax.add_patch(Rectangle((x, y), w, h, color = "red", fill = False))Mask Outline
The mask outline is drawn in the image using _drawMask function with fill = False i.e the contour is not filled and the outline is shown.
def _maskOutline(image, cnts, ax):
img = _drawMask(image, cnts, False)
ax.imshow(img)We generate the heatmap of the entire image by applying COLORMAP_JET and store it in heatmap_img. We draw contours using drawContours from OpenCV and draw outlines or fill the contour area in the image with zeros (markers). Then, we create a mask for the non-zero area in markers and fill the actual image in the masked area with heatmap_img. The third argument -1 in drawContours specifies to draw all the contours and the fifth argument “t” specifies the thickness, -1 filling the entire contour.
def _drawMask(image, cnts, fill=True):
image = np.array(image)
markers = np.zeros((image.shape[0], image.shape[1]))
heatmap_img = cv2.applyColorMap(image, cv2.COLORMAP_JET)
t = 2
if fill:
t = -1
cv2.drawContours(markers, cnts, -1, (255, 0, 0), t)
mask = markers>0
image[mask,:] = heatmap_img[mask,:]
return imageHeatmap
The _drawMask is used in heatmap generating with fill = True (default). We use an interactive widget i.e slider from 0 to 1 with 0.1 increments to show the heatmap at different intensities. We are overlaying the image and heatmap with different intensities by specifying alpha in imshow.
def _heatmap(image, cnts):
fig2 = plt.figure()
plt.axis('off')
hm = st.slider("slider for heatmap", min_value=0.0, max_value=1.0, step=0.1, value=0.5)
img = _drawMask(image, cnts)
plt.imshow(img, alpha=hm)
plt.imshow(image, alpha=1-hm)
plt.title("heatmap")
st.pyplot(fig2)
We then call read_image and show_image from main.
def main():
st.set_page_config(page_title='Omdena Envisionit', page_icon=None, layout='centered', initial_sidebar_state='auto')
st.title('Detecting Pathologies Through Computer Vision in Ultrasound')
image = read_image('image')
mask = read_image('mask')
if image and mask:
show_image(image, mask)We call the main function when the script/module is run.
if __name__ == "__main__":
main()Running the APP
We store the code in a file (app.py) and run the Streamlit app using
streamlit run app.py > streamlit run app.py
You can now view your Streamlit app in your browser.
Local URL: http://localhost:8501
Network URL: http://XXX.XXX.XXX.XXX:8501
Conclusion
We successfully created a Streamlit app that takes the inputs — image and mask by browsing files in the computer, and shows the bounded box and mask outline on the image and a heat map with an interactive slider to get a sense of the tumor intensity.
