Creating and deploying Deep Learning based android and web applications for medical image analysis.
Today we will learn how to create and deploy a medical imaging application using the Google Cloud platform.
This blog is an extension to my previous blog post about Malaria detection using cell images and Deep Convolution Neural networks in Keras! In this blog, we will learn how to create a simple web and android application and deploy our deep learning models in cloud so that we can showcase our work to other people.
I won’t talk much about training the deep learning models since I assumed that you guys are already familiar with that. You can refer to my previous blog post here to get a detailed idea about how I have trained and improved the models using various strategies like Dropouts, Batch Normalization, Optimization, and Data Augmentation. For all the 6 models, I have used a similar set of architectures to train the models with few minor changes here and there.
For those of you who want to jump directly to the code section and skip the rest of this blog, here’s my GitHub repository. For those of you who want to have a taste of this application visit this link.
First things first: What are the features of this app?
Update: The IP address has been updated. To test/view the application please visit this URL (http://104.197.213.77/home)
This android application can detect and categorize 6 diseases — Malaria using cell images, Brain Tumor detection using MRI scans, Pneumonia using chest X-ray images, Retinal diseases using Optical Coherence Tomography images, Nonproliferative Diabetes Retinopathy disease and Invasive Ductal Carcinoma using breast cancer histopathology images. All the models are trained on the publicly available datasets from Kaggle.
Just for fun, I have added a feature called Random Object Detection which uses a pre-trained VGG16 model on ImageNet data. This will return the top 5 class probabilities of all the objects that the model thinks an image belongs to. Here’s how the sample output will look like.
Here’s a short video of the working android app!
Things we will learn in this blog:
- Using the trained models to make predictions on test data.
- Using the python Flask framework to build a simple and interactive web application.
- Deploying the web application on localhost and on a Google cloud VM instance.
- Convert the web application into an android application in the easiest possible way and run it on any android devices.
- Use pre-trained models like VGG16 trained on ImageNet data to predict the top 5 class probabilities of a given input image and display the result on the browser.
This is how our project structure looks like:
Before we dive deep into building the application, let us just have a brief look at the project structure below.
- static folder: This folder is present inside our root folder and will contain all the static files such as css, bootstrap templates, js files, image files, stored model weights that we need throughout the project.
- templates folder: This folder will contain all the HTML files that we will use across the entire project. All the redirects and routing will happen through this folder. This is the default place where Flask looks for rendering templates when we use the render_template() function.
- core_app.py file: This is our main back-end file. We will write all the back-end codes for Flask and python logic in this file. We will run this file to start the server to host our application. This file is basically a link between the HTML files displaying the output on the browser and the model calls.
- index.html file: This is the default web page that is displayed on app load. Hit the enter button to enter the application. Based on your choices from there, the app will redirect you to the specific web pages which will help you analyze different input image results.
Training all the models:
I have used more or less the same architecture for training all our CNN models. Check Malaria detection using cell images and Deep Convolution Neural networks in Keras! to learn how I have trained the models. After the training is completed for all 6 models we have saved them using the model.save() function in Keras. At the end of the whole training, we have obtained six models and saved them in the static/weights/ directory.
First let’s understand the workflow of the backend file.
1. Importing all the Libraries.
In this section we will load all the libraries that we need to build the deep learning application. We will understand how each of them works as we proceed.
2. Instantiating the flask object and routing.
Here we are making an instance of the Flask() class. We will use this instance to map all the incoming requests to other HTML files. If the incoming URL has /home in it, the app will render the file index.html present in the default templates folder. Similarly, if the URL has /checkup, the file checkup.html will get rendered. This is the idea behind routing to different files based on input URLs. If someone wants to go the pneumonia detection application, they can just click on “Pneumonia Detection”. And based on the declarations inside @app.route(), the pnm.html web page will be rendered. For the complete routing lists see the full code in Github.
After this step let’s define a routing function to handle POST and GET requests. Both POST and GET method is used to transfer data between a client and a server. However, POST is more secure since it carries the request parameters inside the message body unlike GET where the request parameters are appended to the URL.
The index() method gets called when a POST request is sent from the client HTML pages to the backend files. The variable ‘type_’ will contain the type value of the request. This value is defined in the HTML file. Look at this code.
Suppose you are at the pneumonia detection page and you want to send upload chest x-ray images, we will pass the value “pnm” to the backend file to make it understand that we will call and predict using the Pneumonia Detection model. This “pnm” value is store in type_. The uploaded image temp directory will be present in request.files[‘img’]. We will then load the image from the temp directory in the variable name.
3. Processing the input images.
For each type_ value, we will process the images differently. For example the Pneumonia Detection model was built using a 128 by 128 pixel image. So we will just write an if-else construct to let the core_app.py file know which processing technique to use for a particular request value parameter contained in type_.
In the above code, we will load the image using PIL library, resize the images to 128 by 128 pixels and convert it to a numpy array. We will normalize the test images by diving all the pixel values by zero. For an RGB image the image dimensions will be 128 x128 x 3. Before feeding it to our model, we will have to add an extra dimension using np.expand_dims() function so that it now has the shape 1 x 128 x 128 x 3. This is how our models were trained and so we have to pre-process the images before predicting the results.
4. Loading the models.
The load_model() function will load the trained models saved at the weights folder inside static. For each request calls, the value will be contained inside type_. The name variable is just a parameter which contains the type values for all the specific models. For Pneumonia Detection model, the type will be “pnm” and accordingly when we want to use the pneumonia detection model the model “pneumonia.h5” will be loaded. The get_model() function will return a single list of dictionary, where the key will the loaded model and the value will be it’s type. For every request, we will load the model using this code below.
4. Making predictions using the loaded models.
After we load the model, we will make prediction on a single image and pass it to the translate_pnm() function. This function will return the class probabilities along with the prediction result. We will then append the results to a json file object. We will then create a response by using the flask function jsonify(final_json). This response will be displayed in the browser. The json object is basically a dictionary which contains predictions and probability scores. Note that there are multiple translate functions defined for multiple models. Here, I am explaining with just one example. For the complete code visit my GitHub repository.
5. Let’s see what the translate functions does.
In the above code example, we are taking the class probabilities into three variables since this model is a 3 class classification model. We are creating a list called total. This variable contains the three class probabilities represented in percentage format. The predictions variable will contain the predicted class label for whichever probability value is highest. We will return these two objects, which is interpreted as preds and pred_value in the above if-else construct. Then we will create a json object as shown in the image present above this image, jsonify the object and send it as a response and use HTML and JavaScript to render the results in a web page. We will see how it’s done.
6. Starting the server.
We will simply use the below code to start the python server.
Let us now understand the frontend section of the application.
The header section.
Once you click “Enter the Application” button defined in the index.html file, you will be re-directed to the checkup.html file. From here if you click on “Pneumonia Detection” you will be re-directed to the pnm.html file.
The head section of pnm.html file will contain references to external JS, CSS files which will be used to render the frontend elements in the web browser.
Reading an input image.
We will take an image as input, and on click of the submit button the sumbit() function defined inside pnm.js will be called. Let’s see what this function does and how it interacts with the backend python file.
Here we are using the XMLHttpRequest object to request data from our web server using the POST method. The variable val will contain “pnm” since we have selected the Pneumonia Detection application. We will initialize a FormData object as data which will be used to capture the predictions made by our model and display them on the webpage. We will parse the JSON response text and display them on the web page using the ids status, status1 and status2. The id result will contain the final predictions. 127.0.0.1 is the localhost IP. We will send requests to localhost for rendering templates and display the elements in the frontent web server. We will learn about this when we talk about deployment. The xhr.send(data) code sends the request to the server. Let’s see where this gets displayed in the HTML file.
Displaying the results in the web page.
Now that we have learned the workflow for our chest pneumonia detection model, the same principals will be applied to all our models. What we will do is just copy paste the codes to create separate JS files and HTML files for each of the models. Again, please visit the GitHub repository for the whole code. Note, that you may find some redundant codes because I have not implemented the other features as of now.
Deploying the application on localhost.
Once all the templates and JS files are ready, let’s use Flask to deploy our application on localhost. For this we just need to go to the root directory and type “python core_app.py” in the terminal and hit enter. This will start our server at port 80. You can use any port as you want. If you have successfully started the server, you will get a message as shown below.
After this step, let’s open any web browser from our PC and type:
htpp://127.0.0.1:80/homeIf you have done everything correctly till now, you are expected to land up in the index.html page as shown below.
On clicking the “Enter the Application” button, you will land up in the checkup.html page where you are expected to see the below web page.
From here, lets click on Chest Pneumonia Analysis and see what happens next, i.e. the page where the Pneumonia Detection application will run. We will click on browse and upload an image. On clicking Analyse Image, the corresponding JS functions will get called, which will then interact with the backend files and render the outputs in the web page as shown below.
Deploying the application on Google cloud.
After we are completely sure our application is running fine in localhost, we will upload it into a GitHub repository. We will clone this repository in a google cloud virtual machine instance before deploying the application. But first, let us understand how to use the Google cloud platform.
Before we get started with hosting, let me make it very clear that you need Google Cloud credits to create any VM instance. If you don’t have Google Cloud credits, it’s not an issue. Just search on Google “How to create Google actions”. Once you create a Google action, Google will reward you with $200 credits every month for the next one year. $200 per month is a lot! You can literally create high end VM instances with that amount of money. Now considering you have created a Google action and you have sufficient credits let’s see how to create a Google VM instance.
Setting up a Google Cloud VM instance.
- Login to this URL using your Gmail ID and you hit the “Go to console” button.
2. Once you have clicked the “Go to console” button, you will land up in your own dashboard page. Click the navigation menu on the top left corner, go to “Compute Engine” and then “VM instances”. Look below for reference.
3. You will land up in a page like the one shown below.
4. Click on “CREATE INSTANCE”. Chose your virtual machine configuration in the dropdown “Machine type”. Scroll down, look at the firewall section and select “Allow HTTP and HTTPS traffic.” Click on the create instance button and Google will create an instance for you. Not this IP address. We will need this when we deploy our application.
5. Once you have created an instance you can connect to its terminal using the SSH service. Click on SSH and then click on “Open in browser window”
6. A Linux terminal will open in the browser like the one shown below, provided you have performed all the steps correctly.
Deploying the web application on Google cloud.
- The first step that you need to perform is get the IP of the VM instance. You can type ifconfig in the terminal or you can simply check the IP displayed beside your VM instances.
- Remember previously we have deployed our application in localhost where we had given the IP 127.0.0.1 in the JS files to handle all the POST requests? We just need to change all the IPs in all the JS files to the IP of your VM instance. Look at an example of how to change the IPs.
3. After you have made the necessary changes, push your code to your GitHub repository however you want to. For simplicity you can play around with my code hosted at GitHub to try this out.
4. Open the SSH terminal and clone the repository by typing the following command:
git clone https://github.com/saugatapaul1010/Medical-AI-AndroidIf your VM instance does not have git installed please install git here. This will download all the files present in the repository to your VM instance.
5. First, we need to navigate to the working directory and see the contents inside this folder.
cd Medical-AI-Android/WebApp
ls6. You can see the file requirements.txt file present inside this directory. It contains all the packages that we need to run this application. Install all the packages by typing in the console:
sudo pip install -r requirements.txt7. Now wait for all the packages to get installed. Once this step is done, we need to run our application by typing:
sudo python core_app.pyThis will start the server in the Google Cloud VM instance. To access the Web Application just type 35.202.235.200:80/home in your PC browser and hit enter. You will be greeted with the application home page. Needless to say, 35.202.235.200 is the IP of my VM Machine.
8. Once your application is successfully deployed on Google Cloud you see a page like below. Notice the URL, the application is not running on localhost anymore. It’s hosted in Google Cloud VM instance.
A short video of the application deployed on Google Cloud:
Creating an android application from the web application.
- Well, for people who do not know how to code on Android frameworks (like me), it doesn’t mean you can’t build android apps! All you need to do is to have a web application ready and we will convert it to an Android APK by using a free software called WebsiteToApk. You can download the software here. This software is available only for Windows machines. Since I use Linux, I have borrowed by brothers laptop to create the APK file. Let’s go through the steps. It’s fairly straight forward.
- Remember we have changed all the IPs in the JS files to our Google Cloud IP? Let it be that way. This software needs all the HTML files to be present in the root directory. Just copy paste the website that we have built and structure it as follows. Note, that the IP present in the JS files should be that of you Google Cloud instance.
3. Run the application in an Windows PC and you should be greeted with the following screen.
4. Chose “Local HTML Website” in the Website to convert section. Chose a desired app name. Chose the folder structure that we have created earlier. Chose an output directory for your apk file and then click on the GENERATE APK button. Voila! Your android APK file will be ready in no time.
5. Copy this APK file in your phone and install it. The APK file will connect to the hosted server on Google Cloud using the IP it is configured to use. Here’s a sample video of the android application I have recorded using my phone.
Random Object Detection model.
In this section we will talk about how to use a pre-trained VGG16 model trained on ImageNet data and fit the model onto our web and android application.
- First we need to get the trained VGG16 model weights from Keras. You need to rename the downloaded file to vgg16model.h5 and save it to the /static/weights directory. You can download the model by typing:
wget https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h52. After the weight file is renamed and saved in the desired folder, we need to load the model weights. Remember, simply using load_model() function from Keras won’t work this time. You need to instantiate the VGG16 model architecture before loading the weights and making predictions using it. Let’s define a function for doing this task.
3. For any input image, we need to pre-process it using the VGG16 ImageNet utils. This is needed to ensure that the network sees the image in that specific format in which it was trained.
4. After processing the input image, we will load the VGG16 pre-trained model by calling the function load_vgg16_model(). Once we make a prediction using the input image, we will pass these parameters to the translate_vgg_16() function. The workflow of this model is exactly the same as discussed in the Backend and Frontend work flow sections.
5. Here is the code snippet for the translate_vgg_16() function definition. The function decode_predictions() will by default return the top 5 class labels along with it’s probability score. The variable total will contain the top 5 class probabilities of the input image and the corresponding class labels.
6. We have used convert_results() function to format the output strings according to our need.
7. The output of this will be passed as a response to the final web/android page via the JSON object and the JS files.
References:
- A huge thanks to my brother Suvhradip Ghosh! Without him, the UI would not have been possible.
- Thanks to Jeremy Chow for his post on deployment of Machine Learning models using Flask.
- Thanks to the team at AppliedAICourse for teaching me Deep Learning.
- The images and GIFs used in this application may be subjected to copyrights. They are used from the following URLS:
- https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQDJlkDxXnF-FUBBZH9ChtKrOtqJO1CDgwvisA6vLhyaJHBIt0R
- https://media1.giphy.com/media/VGK2WUT3amXjG/giphy.gif
- https://thumbs.gfycat.com/AnxiousThinKite-small.gif
- http://5b0988e595225.cdn.sohucs.com/images/20180910/2c8f268530144710a79165fec612122b.jpeg
- http://raiingkaeng.com/images/dark-blue-wallpaper/37216562-dark-blue-wallpaper.jpg
5. The Bootstrap templates that we have used in our application can be found in this URL: https://bootstrapmade.com/devfolio-bootstrap-portfolio-html-template/
6. All the datasets are taken from Kaggle.
Future Work.
- We are planning to build a heart beat detector using a CNN-RNN architecture and integrate it with this application.
- The future updates will also see on how to detect and interpret ECG signals.
- We will keep building this project and add more diseases as we move along. Our primary focus for now is to improve the brain tumor model with much more data samples.
- We will implement GPS based hospital recommendation system using the Google Maps API. We are not yet finished with the whole code.
If you like this blog and the application please use our code, modify it and suggest changes that we need to make in order to make this application more stable and with many more features.
If you are someone who would like to collaborate with me for this project, please feel free to reach me at saugata.paul1010@gmail.com.
Here’s the full code at GitHub.
Here’s the link to the application: http://104.197.213.77/home
