How to build real-time object recognition iOS app

In this article, I will introduce how to build your own real-time object recognition iOS app. Thanks to other people’s articles you can easily train your own object recognition model using Tensorflow Object Detection API and integrate the trained model into your iOS app.

This is my real-time Sushi recognition iOS app:

Motivation

When Google released Tensorflow Object Detection API, I was really excited and decided to build something using the API. I thought a real time object detection iOS (or Android) app would be awesome.

Why would I want to recognize objects in real time?

Of course, you can host a remote API which detects objects from a photo. But sometimes you don’t want to wait for the latency, sometimes you don’t have internet. When you want to know what objects in front of you and where immediately, detecting objects in your camera screen would be useful.

Ok …, but why sushi? 🍣🍣

Because everone likes sushi. (this is what my colleague told me 😋)
Seriously, It would be awesome if an app enables people to access expert knowledges.
In my case, types of sushies. Some sushes look similar, some look totally different. So it would be a good test case to know the capability of real-time object detector.

Train your own object detection model

After reading few blog posts, I was able to train my own object detection model. I won’t explain the details of this step in thie article, please check my Github repository and/or the official document for more details.

Export trained model

After finishing trainig, you need to export your model. Follow this document for more details. In the output directory, you will find frozen_inference_graph.pb, which will be used on iOS later.

Build an iOS app using the trained model

Even though the accuracy of the trained model wasn’t perfect, I wanted to run it on mobile and see how it works. Then I started to find how to integrate tensorflow model into iOS app.

I found this repository which demonstrates an object detection app on iOS using Tensorflow and pre-trained datase models. I will explain the steps I did to integrate my custom trained model into the iOS app project.

Build Tensorflow libraries for iOS

I had to build Tensorflow libraries for iOS on my machine. This step took few hours.

brew install automake libtool
git clone https://github.com/tensorflow/tensorflow && cd tensorflow
git checkout v1.5.0
export ANDROID_TYPES="-D__ANDROID_TYPES_FULL__"
tensorflow/contrib/makefile/build_all_ios.sh

After finishing the build step, I changed tensorflow.xconfig where Tensorflow path is configured.

Integrate your own model

You can see all changes I did in my Github repo.

Frozen model

After removing existing models from the project, I placed the frozen model I created before in the project and added to iOS project as a resource named op_inference_graph.pb. You can also change the model name by changing code.

Label map

I also had to add a label map which was used in the training step. But I also added display_name. And I changed the label map name in the code.

item {
    id: 1,
    name: 'salmon',
    display_name: 'salmon'
}

Result

After the above steps, I was able to build an iOS app successfully.

Video:

Speed:
Predictions are done in 0.1 sec on average.

...
Avg. prediction time: 0.121424
Avg. prediction time: 0.118740
Avg. prediction time: 0.125094
...

Conclusion

I hope you enjoyed this post and you can now build your own real-time object detector app on iOS. In this post, I didn’t care about app size, prediction speeds. As next steps, I want to try to use Tensorflow Lite to reduce app size and speed up inferences. Google announced that Tensorflow Lite supports iOS’s CoreML converting Tensorflow model to CoreML model. I’m looking foward to report the results later.

Thank you for reading this post, please give me a ❤️ if you liked.