Open in app

Sign in

Write

Sign in

Tutorial: Real-time Android Object Detection of Pneumonia Chest X-Ray Opacities using SSD Mobilenet V1

Daniel Fleury
8 min readOct 28, 2018

--

Real-time Neural Net and Object Detection Status on Android Pneumonia Detector

Chest X-Ray Opacities are critical in early-stage diagnosis and confirmation of child-adulthood Pneumonia. Broadly, Pneumonia is associated with 15% of all deaths of children under 5 years old on a global scale. Further, Pneumonia accounted for over 500,000 clinical-emergency hospital interventions and ~50,000 deaths in 2015. Although

Pneumonia Chest X-Ray with Rising Opacity in the patient’s right lung (University of Virginia Medical Department )

The Tensorflow object detection API library and its model zoo have expanded to configure training and deployment on varying levels of scalability. More importantly, neural networks have diversified from models such as Faster R-CNN to another end of the spectrum such as SSD Mobilenet. Drawbacks become apparent in terms of real-time capabilities and broad precision/accuracy (i.e. Faster R-CNN provides higher accuracy while sacrificing immediate visualizations and real-time detections at high FPS on small-scale devices.

This tutorial is a comprehensive and straightforward block-by-block process of implementing SSD Mobilenet towards the real-time detection of Pneumonia opacities. The first bulk of this protocol will include testing still images through an OpenCV Python module, while the second half will deploy the frozen_inference_graph.pb on a prepackaged Tensorflow Android detector. The most crucial components of this tutorial include:

Overview

  1. Installing Anaconda and initiating a virtual environment.
  2. Install Tensorflow and associated library dependencies.
  3. Downloading both the Tensorflow Object Detection API Repository and tutorial repository (EdjeElectronics’ Faster R-CNN Repository).
  4. Download SSD Mobilenet V1 Coco Model
  5. Set Python Paths and Compile Protobuf Directory
  6. Jupyter Notebook Testing and Diagnosis of the Object Detection Library
  7. Image Annotation with LabelImg
  8. Preparing Training Data and Generating .record file and training the model.
  9. Testing model on OpenCV Module.
  10. Android Deployment

___________________________________________________________________

Anaconda Installation and Setting up a Virtual Environment

The Windows Anaconda (version 5.3) with Python 3.7 can be easily downloaded here (https://www.anaconda.com/download/). All command line facets of this tutorial will be implemented on the Anaconda command prompt. Additionally, a virtual environment that defines all essential Python paths and Tensorflow/project dependencies are necessary.

Once Anaconda has been successfully installed, establish a new virtual environment with the following command:

C:\> conda create -n tensorflow pip python=3.7

Finally, activate the virtual environment by executing:

C:\> activate tensorflow

Install Tensorflow GPU and Additional Libraries

An intensive Tensorflow object detection task requires massive NVIDIA Cuda GPU utilization, so issue:

GPU

(tensorflow) C:\> pip install --ignore-installed --upgrade tensorflow-gpu

Finally, download all other necessary package dependencies by executing these commands:

(tensorflow) C:\> conda install -c anaconda protobuf
(tensorflow) C:\> pip install pillow
(tensorflow) C:\> pip install lxml
(tensorflow) C:\> pip install Cython
(tensorflow) C:\> pip install jupyter
(tensorflow) C:\> pip install matplotlib
(tensorflow) C:\> pip install pandas
(tensorflow) C:\> pip install opencv-python

Alternatively, use pip install -r requirements.txt which is referenced in the object detection folder by issuing:

(tensorflow) C:\tensorflow\models-master\research\object_detection> pip install -r requirements.txt

Download both the Object Detection API Repository and the Generic Tutorial repository:

Once both repositories are downloaded drag all EdjeElectronics files/folders into the object detection folder of the API directory. After you’ve unified files into the API directory delete the following in \object_detection\

  • Both csv files in the “images” folder.
  • All files in images\train and images\test
  • All files in \training
  • Al files in \inference_graph

Download the SSD Mobilenet V1 Folder from Tensorflow’s Model Zoo

Tensorflow’s object detection API provides compatibility for a variety of pretrained models, including the Mobilenet series, Faster R-CNN, and Mask R-CNN. However, this object detection task requires high FPS rates. Download and extract the SSD Mobilenet V1_coco model here:

http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz

Following extraction, drag the SSD Mobilenet v1 Folder into the \object_detection folder.

Configure Python Paths and Compile Protobufs:

Set the following python paths pointing to the slim directory in the object detection API folder:

(tensorflow) C:\> set PYTHONPATH=C:\tensorflow\models;C:\tensorflow\models\research;C:\tensorflow\models\research\slim

Compile and Convert all .protoc files from the research directory:

protoc --python_out=. .\object_detection\protos\anchor_generator.proto .\object_detection\protos\argmax_matcher.proto .\object_detection\protos\bipartite_matcher.proto .\object_detection\protos\box_coder.proto .\object_detection\protos\box_predictor.proto .\object_detection\protos\eval.proto .\object_detection\protos\faster_rcnn.proto .\object_detection\protos\faster_rcnn_box_coder.proto .\object_detection\protos\grid_anchor_generator.proto .\object_detection\protos\hyperparams.proto .\object_detection\protos\image_resizer.proto .\object_detection\protos\input_reader.proto .\object_detection\protos\losses.proto .\object_detection\protos\matcher.proto .\object_detection\protos\mean_stddev_box_coder.proto .\object_detection\protos\model.proto .\object_detection\protos\optimizer.proto .\object_detection\protos\pipeline.proto .\object_detection\protos\post_processing.proto .\object_detection\protos\preprocessor.proto .\object_detection\protos\region_similarity_calculator.proto .\object_detection\protos\square_box_coder.proto .\object_detection\protos\ssd.proto .\object_detection\protos\ssd_anchor_generator.proto .\object_detection\protos\string_int_label_map.proto .\object_detection\protos\train.proto .\object_detection\protos\keypoint_box_coder.proto .\object_detection\protos\multiscale_anchor_generator.proto .\object_detection\protos\graph_rewriter.proto

Test and Troubleshoot: Object Detection Notebook Demo

GIven that all libraries and dependencies are installed, the object detection API is prepared to utilize GPU and load a frozen model to output a generic image with bounding boxes. Issue the following command to enter Jupyter Notebook:

(tensorflow) C:\tensorflow\models\research\object_detection> jupyter notebook object_detection_tutorial.ipynb

Run each cell individually and respond/troubleshoot any error messages that appear below the cell (searching online to resolve and understand import errors, etc, is critical). If the scenario follows through, the following bounding box outputs should conclude the last cell:

Prepare Training Data:

You can access an ~25,000 image dataset of Pneumonia chest x-rays from the Kaggle RSNA Pneumonia Detection challenge:

https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/data

Label training images with the LabelImg Annotation Tool. Download LabelImg here (https://www.dropbox.com/s/tq7zfrcwl44vxan/windows_v1.6.0.zip?dl=1)

Once downloaded, label images as represented below. For bare-minimum detection, at least ~100 images are necessary. Use the “open dir” to open your folder of selected images, and “change save dir” to output all XML files in the desired directory. Once all XML files are outputted, 80% of the images (paired with XML files) into the images\train and 20% of the data into images\test

Generate TF-Record Training Data:

Converting all output XML files into an interpretable train and test CSV file is necessary to generate .record training files. Issue the following command:

(tensorflow) C:\tensorflow\models\research\object_detection> python xml_to_csv.py

..which outputs train_labels.csv and test_labels.csv in the \images folder.

Open the generate_tfrecord.py file and modify the row_labels to:

# TO-DO replace this with label map
def class_text_to_int(row_label):
    if row_label == 'opacity':
        return 1
    else:
        return 0

Issue the following commands from the object_detection directory to generate both train.record and test.record.

python generate_tfrecord.py --csv_input=images\train_labels.csv --image_dir=images\train --output_path=train.recordpython generate_tfrecord.py --csv_input=images\test_labels.csv --image_dir=images\test --output_path=test.record

The configuration file and the OpenCV module requires a labelmap to annotate the label boxes. In the \training folder create the file “labelmap.pbtxt” containing:

item {
  id: 1
  name: 'opacity'
}

It is necessary to modify the SSD Mobilenet v1 Config file. First, navigate into the C:\tensorflow\models\research\object_detection\samples\configs and drag the ssd_mobilenet_v1_pets.config file into the \object_detection\training folder. Modify the following lines:

  • Line 9- num_classes: 1
  • Line 110- fine_tune_checkpoint: C:/tensorflow/models/research/object_detection/ssd_mobilenet_v1_coco_2017_11_17/model.ckpt
  • Lines 126 and 128- train_input_reader → input_path : “C:/tensorflow/models/research/object_detection/train.record”
  • label_map_path: “C:/tensorflow/models/research/object_detection/training/labelmap.pbtxt”
  • Line 132- num_examples: #number of images in the \images\test directory
  • Line 140 and 142- eval_input_reader section input_path : “C:/tensorflow/models/research/object_detection/test.record”
  • label_map_path: “C:/tensorflow/models/research/object_detection/training/labelmap.pbtxt”

Finally, Execute Training

from the object_detection directory, issue:

(tensorflow) C:\tensorflow\models-master\research\object_detection>python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config

The training process should commence and loss values should be reported as shown below:

Open a separate Anconda command prompt and open Tensorboard to visualize scalars from the object detection directory. Tensorboard will probably be hosted on a DESKTOP link, so navigate to that link on your web browser.

Tensorflow will save .ckpt files in the \training folder

(tensorflow)C:\tensorflow1\models\research\object_detection>tensorboard --logdir=training

Export the Inference Graph:

After loss values have converged around ~2.00 on the classification-localization loss scalar, it is necessary to export the inference graph for final bounding box testing.

I have periodically exported inference graphs in steps ranging from 5,000–20,000 steps. Open the export_inference_graph.py file in the \object_detection folder and simply modify the following to the most recently saved .ckpt file in the \training directory.

For example, my training folder appears as the following:

The latest saved .ckpt file is at 9738 steps, so I would open the export_inference_graph.py in the \object_detection directory and change the following line from:

flags.DEFINE_string(‘trained_checkpoint_prefix’, ‘training/model.ckpt-xxxx’,
 ‘Path to trained checkpoint, typically of the form ‘
 ‘path/to/model.ckpt’)

to

flags.DEFINE_string(‘trained_checkpoint_prefix’,    ‘training/model.ckpt-9378’,
 ‘Path to trained checkpoint, typically of the form ‘
 ‘path/to/model.ckpt’)

Save the resulting frozen_inference_graph.py file and run it from the command line:

(tensorflow)C:\tensorflow1\models\research\object_detection>python frozen_inference_graph.py

The frozen_inference_graph.pb will be outputted in the object_detection\inference graph folder. In Anaconda, type “idle” to open the Python shell module. Open Object_detection_image.py set change NUM_CLASSES=1 and simply change the test image (IMAGE_NAME = ) to the name of the image in the \object_detection folder. Modifying the relevant fields yields the following bounding box results:

The frozen_inference_graph.pb model is operating with high confidence values. We can now prepare the file for Android deployment.

Real-time Android Detector Deployment:

Download and extract the entire Tensorflow Github repository here: https://github.com/tensorflow/tensorflow

Following extraction, navigate to the \tensorflow\examples\android\assets directory. Copy and Paste your frozen_inference_graph.pb file from the inference_graph folder in your object_detection API directory. Create and save a labels.txt file containing the following:

???
opacity

Finally, assuming you’ve downloaded Android studio, file<open \tensorflow\examples\android. In left-side navigation panel, open the java<org.tensorflow.demo<DetectorActivity.java file. The file uses the Tensorflow Object Detection API by default. In lines 66–67 replace:

“file:///android_asset/ssd_mobilenet_v1_android_export.pb”; private static final String TF_OD_API_LABELS_FILE = “file:///android_asset/coco_labels_list.txt”;

with

“file:///android_asset/frozen_inference_graph.pb”; private static final String TF_OD_API_LABELS_FILE = “file:///android_asset/labels.txt”;

Connect your Android device to the computer and enable USB debugging. Once you build the Android APK, it will install all Tensorflow demo applications. Open the “TF Detect” app and test it out! The following video exhibits some test results. You can click your volume buttons to view real-time function processes and neural net layer operations:

The following references were essential in my own personal research and in the composition of this tutorial:

Python
Machine Learning
Artificial Intelligence
Object Detection
Android App Development

--

--

Daniel Fleury

Written by Daniel Fleury

41 Followers

I deploy scalable Deep Learning (DL) algorithms for Computer Vision-driven medicine and public policy | ML and Cognitive Science @ Johns Hopkins University.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams