Object detection with TensorFlow on Raspberry Pi

8 min readAug 5, 2020

The following post shows how to train and test TensorFlow and TensorFlow Lite models based on SSD-architecture (to get familiar with SSD follow the links in the «References» down below) on Raspberry Pi.

Note: The described steps were tested on Linux Mint 19.3 but shall work on Ubuntu and Debian.

Data preparation

Like in the post dedicated to YOLO one have to prepare data first. Follow the first 7 steps and then do this:

1. In order to get listed data and generate TFRecords clone repository «How To Train an Object Detection Classifier for Multiple Objects Using TensorFlow
(GPU) on Windows 10»:

git clone https://github.com/EdjeElectronics/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10.gitcd TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-
Windows-10

2. Put all labeled images into folders «images/test» and «images/train»:

3. Get data records:

python3 xml_to_csv.py

This command creates «train_labels.csv» and «test_labels.csv» in «im-
ages» folder:

4. Open «generate_tfrecord.py»:

And replace the label map starting at line 31 with your own label map, where each object is assigned an ID number, for ex.:

5. Generate TFRecords for data:

python3 generate_tfrecord.py — csv_input=images/train_labels.csv
 — image_dir=images/train — output_path=train.recordpython3 generate_tfrecord.py — csv_input=images/test_labels.csv
 — image_dir=images/test — output_path=test.record

These commands generate «train.record» and «test.record» file which will be used to train the new object detection classifier.

6. Create a label map. The label map defines a mapping of class names to class
ID numbers, for ex.:

item {
 id: 1
 name: 'nutria'
}

Save it as «labelmap.pbtxt».

7. Configure the object detection training pipeline. It defines which model and what parameters will be used for training.
Download «ssd_mobilenet_v2_quantized_300x300_coco.config» from https://github.com/tensorflow/models/tree/master/research/object_detection/samples/configs:

wget https://github.com/tensorflow/models/blob/master/research/object_detection/samples/config/ssd_mobilenet_v2_quantized_300x300_
coco.config

8. Change configuration file:

Set number of classes:
- num_classes: SET_YOUR_VALUE
Set checkpoint:
- fine_tune_checkpoint: “/path/to/ssd_mobilenet_v2_quantized/model.ckpt”
Set «input_path» and «label_map_path» in «train_input_reader»:
- input_path: “/path/to/train.record”
- label_map_path: “/path/to/labelmap.pbtxt”
Set «batch_size» in «train_config»:
- batch_size: 6 (OR SET_YOUR_VALUE)
Set «input_path» and «label_map_path» in «eval_input_reader»:
- input_path: “/path/to/test.record”
- label_map_path: “/path/to/labelmap.pbtxt”

Setup environment

General settings for Raspberry Pi

1. Update and upgrade first:

sudo apt update
sudo apt dist-upgrade

2. Install some important dependencies:

sudo apt update
sudo apt install -y joe telnet nmap htop sysbench iperf bonnie++ iftop nload hdparm bc stress python-dev python-rpi.gpio wiringpi stress sysstat zip locate nuttcp attr imagemagick netpipe-tcp netpipe-openmpi git libatlas-base-dev libhdf5-dev libc-ares-dev libeigen3-dev build-essential libsdl-ttf2.0-0 python-pygame festival

3. Install dependencies for TensorFlow:

sudo apt update
sudo apt install libatlas-base-dev python-tk virtualenv
sudo pip3 install pillow Pillow lxml jupyter matplotlib cython numpy pygame

4. Install dependencies for OpenCV:

sudo apt update
sudo apt install libjpeg-dev libtiff5-dev libjasper-dev libpng12-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev qt4-dev-tools libatlas-base-dev

5. Install OpenCV itself:

sudo apt update
sudo pip3 install opencv-python

6. Install TensorFlow by downloading «wheel» from https://github.com/lhelontra/tensorflow-on-arm/releases:

sudo apt update
sudo pip3 install tensorflow-2.2.0-cp37-none-linux armv7l.whl

Note: Experience shows that it is better to install «wheel» rather then from pip default repository, since it does not show all the versions for Raspberry Pi:

Training

Note: Training shall be done on host machine to avoid additional problems that might occur on Raspberry Pi since TensorFlow framework and its accompanying software were originally developed and optimized for usage on mainframes.

1. Install TensorFlow (for CPU or GPU):

sudo pip3 install tensorflow==1.13.1
or
sudo pip3 install tensorflow-gpu==1.13.1

Note: Use v1.13.1 since it is the most stable version for main frames and works with all other software used here (from own experience).

2. Get TensorFlow models:

git clone https://github.com/tensorflow/models.git

3. Copy «train.py» from folder «legacy» to «object_detection»:

cp /path/to/models/research/object_detection/legacy/train.py
/path/to/models/research/object_detection/

4. Get pretrained model from https://github.com/tensorflow/models/blob/
master/research/object_detection/g3doc/tf1_detection_zoo.md:

wget http://download.tensorflow.org/models/object_detection/ssd_
mobilenet_v2_quantized_300x300_coco_2019_01_03.tar.gz

5. Unpack archive:

tar -xvzf ssd_mobilenet_v2_quantized_300x300_coco_2019_01_03.tar.gz
-C /destination/folder/

Note: Unpack archive in folder for which «fine_tune_checkpoint» is configured in «*.config».

6. Start training:

python3 train.py --logtostderr -train_dir=/path/to/training/
--pipeline_config_path=/path/to/ssd_mobilenet_v2_quantized.config

Note #1: «/path/to/training/» is any folder where all training results could
be saved to.
Note #2: If training process is suddenly terminated one can change values
«num_steps» and «num_examples» reducing the load on memory.

7. After training has finished, the model can be exported for conversion to TensorFlow Lite using the «export_tflite_ssd_graph.py» script:

python3 export_tflite_ssd_graph.py
--pipeline_config_path=/path/to/ssd_mobilenet_v2_quantized.config
--trained_checkpoint_prefix=/path/to/training/model.ckpt-XXXX
--output_directory=/path/to/output/directory
--add_postprocessing_op=true

Note #1: For each «model.ckpt-XXXX» there must be corresponding «model.ckpt-XXXX.data-00000-of-00001», «model.ckpt-XXXX.index», «model.ckpt-XXXX.meta” in the «training» folder.
Note #2: «/path/to/output/directory» is any folder where all final results could be saved to.

After the command has been executed, there must be two new files in the
output folder specified for «output_directory»: «tflite_graph.pb» and
«tflite_graph.pbtxt».

8. Install Bazel in order to optimize trained model through the TensorFlow Lite Optimizing Converter (TOCO) before it will work with the TensorFlow Lite interpreter:

Install dependencies:

sudo apt install g++ unzip zip
sudo apt install openjdk-11-jdk

Download version 0.21.0 (from https://github.com/bazelbuild/bazel/
releases/tag/0.21.0):

wget https://github.com/bazelbuild/bazel/releases/download/
0.21.0/bazel-0.21.0-installer-linux-x86_64.sh

Note: The experience shows that only Bazel v0.21.0 works well. Other versions cause multiple errors.

Change permission rights:

chmod +x bazel*.sh

Install Bazel:

./bazel*.sh –user

Installation is shown for Ubuntu (https://docs.bazel.build/versions/
master/install-ubuntu.html). The same steps are applicable for Debian and Linux Mint. For other OS follow installation guide from
https://docs.bazel.build/versions/master/install.html

9. Clone TensorFlow repository and open it:

git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow

10. Use Bazel to run the model through the TOCO tool by issuing this command:

bazel run --config=opt tensorflow/lite/toco:toco --
--input_file=/path/to/tflite_graph.pb
--output_file=/path/to/detect.tflite
--input_shapes=1,300,300,3
--input_arrays=normalized_input_image_tensor
--output_arrays=TFLite_Detection_PostProcess,
TFLite_Detection_PostProcess:1,
TFLite_Detection_PostProcess:2,
28TFLite_Detection_PostProcess:3
--inference_type=QUANTIZED_UINT8
--mean_values=128
--std_values=128
--change_concat_input_ranges=false
--allow_custom_ops

Note: The output could be the following:

After the command finishes running, there shall be a file called «detect.tflite» in the directory specified for «output_file».

11. Create «labelmap.txt» and add all class (object) names for which the model was trained:

touch labelmap.txt

The contents:

12. The model is ready for usage. Put «detect.tflite» and «labelmap.txt» into separate folder and use it as normal pretrained model (see «Testing» paragraph).

Testing

For TensorFlow Lite model

For custom model

1. Clone repository for Raspberry Pi and open it

git clone https://github.com/EdjeElectronics/TensorFlow-Lite-Object\
-Detection-on-Android-and-Raspberry-Pi.git
cd TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi

2. Put earlier trained model (custom «detect.tflite» and «labelmap.txt») into «/path/to/model» and run the command:

python3 /path/to/TensorFlow-Lite-Object-Detection-on-Android-and-
Raspberry-Pi/TFLite_detection_webcam.py –modeldir=/path/to/model

For pretrained model

The same is applicable to already pretrained model.

1. Download pretrained SSD MobileNet from https://www.tensorflow.org/lite/models/object_detection/overview:

wget https://storage.googleapis.com/download.tensorflow.org/models/
tflite/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip

2. Unzip the model:

unzip /path/to/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip -d
/path/to/model

Archive must contain «detect.tflite» and «labelmap.txt» files.

3. Open cloned repository and run the same command:

python3 /path/to/TensorFlow-Lite-Object-Detection-on-Android-and-
Raspberry-Pi/TFLite_detection_webcam.py –modeldir=/path/to/model

For TensorFlow model

1. Install package «argparse»:

sudo pip3 install argparse

2.1. Either copy the script «Object_detection_webcam.py» from «TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10» repository to «models» repository in «/path/to/models/research/object_detection» and add the following:

import package argparse

import argparse

add the following arguments:

ap = argparse.ArgumentParser(description='Testing tools')
ap.add_argument('-pb', '--path_to_pb')
ap.add_argument('-l', '--path_to_labels')
ap.add_argument('-nc', '-num_classes')
args = vars(ap.parse_args())

Comment out lines with variables «MODEL_NAME», «PATH_TO_CKPT»,
«PATH_TO_LABELS», «CWD_PATH» and «NUM_CLASSES» and add the :

ap = argparse.ArgumentParser(description='Testing tools')
ap.add_argument('-pb', '--path_to_pb')
ap.add_argument('-l', '--path_to_labels')
ap.add_argument('-nc', '--num_classes')
args = vars(ap.parse_args())# Name of the directory containing the object detection module we're using
#MODEL_NAME = 'inference_graph'# Grab path to current working directory
#CWD_PATH = os.getcwd()# Path to frozen detection graph .pb file, which contains the model that is used
# for object detection.
#PATH_TO_CKPT = os.path.join(CWD_PATH,MODEL_NAME,'frozen_inference_graph.pb')
PATH_TO_CKPT = args['path_to_pb']# Path to label map file
#PATH_TO_LABELS = os.path.join(CWD_PATH,'training','labelmap.pbtxt')
PATH_TO_LABELS = args['path_to_labels']# Number of classes the object detector can identify
#NUM_CLASSES = 6
NUM_CLASSES = int(args['num_classes'])

2.2. Or download already modified script:

cd /path/to/models/research/object_detection
wget https://bitbucket.org/ElencheZetetique/fixed_scripts/src/master/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10/Object_detection_webcam.py

3.1. Open script «label_map_util.py» in «/path/to/models/research/object_detection/utils/» and either comment out if-statement for «item.keypoints» or add an exception for it:

# if item.keypoints:
      #   keypoints = {}
      #   list_of_keypoint_ids = []
      #   for kv in item.keypoints:
      #     if kv.id in list_of_keypoint_ids:
      #       raise ValueError('Duplicate keypoint ids are not allowed. Found {} more than once'.format(kv.id))
      #     keypoints[kv.label] = kv.id
      #     list_of_keypoint_ids.append(kv.id)
      #   category['keypoints'] = keypoints
      try:
        if item.keypoints:
          keypoints = {}
          list_of_keypoint_ids = []
          for kv in item.keypoints:
            if kv.id in list_of_keypoint_ids:
              raise ValueError('Duplicate keypoint ids are not allowed. Found {} more than once'.format(kv.id))
            keypoints[kv.label] = kv.id
            list_of_keypoint_ids.append(kv.id)
          category['keypoints'] = keypoints
      except AttributeError:
        pass

3.2. Alternatively one might download modified script:

cd /path/to/models/research/object_detection/utils/
wget https://bitbucket.org/ElencheZetetique/fixed_scripts/src/master/models_TF/label_map_util.py

For custom model

1.1. Open script «export_inference_graph.py» in «/path/to/models/research/object_detection» and comment out last parameters:

exporter.export_inference_graph(
      FLAGS.input_type, pipeline_config, FLAGS.trained_checkpoint_prefix,
      FLAGS.output_directory, input_shape=input_shape,
      write_inference_graph=FLAGS.write_inference_graph,
      additional_output_tensor_names=additional_output_tensor_names,
      #use_side_inputs=FLAGS.use_side_inputs,
      #side_input_shapes=side_input_shapes,
      #side_input_names=side_input_names,
      #side_input_types=side_input_types)
      )

1.2. Or copy the script replacing the original one:

cd /path/to/models/research/object_detection
wget https://bitbucket.org/ElencheZetetique/fixed_scripts/src/master/models_TF/export_inference_graph.py

2. Export inference graph using script «export_inference_graph.py» from
«/path/to/models/research/object_detection»:

python3 export_inference_graph.py
--input_type image_tensor
--pipeline_config_path /path/to/ssd_mobilenet_v2_quantized.config
--trained_checkpoint_prefix /path/to/training/model.ckpt-XXX
--output_directory /path/to/output/directory

3. In the output directory assigned for flag --output_directory there
must be file «frozen_inference_graph.pb»:

4. Run modified script «Object_detection_webcam.py» for custom model:

python3 Object_detection_webcam.py -nc 1
-pb /path/to/frozen_inference_graph.pb
-l /path/to/labelmap.pbtxt

Example of detection:

For pretrained model

1. Download the model you are interested in from https://github.com/
tensorflow/models/blob/master/research/object_detection/g3doc/
tf1_detection_zoo.md

wget http://download.tensorflow.org/models/object_detection/
faster_rcnn_resnet101_coco_2018_01_28.tar.gz

2. Extract file «frozen_inference_graph.pb» from archive

3. Run modified script «Object_detection_webcam.py» for pretrained model:

python3 Object_detection_webcam.py -nc 100 
-pb /path/to/frozen_inference_graph.pb
-l /path/to/mscoco_label_map.pbtxt

Examples of detection:

Assign maximum number of classes for flag -nc/--num_classes
Assign path to «/path/to/models/research/object_detection/data/mscoco_label_map.pbtxt» for flag -l/--path_to_labels