HowTo Build Tensorflow Apps for RICOH THETA

Build Tensorflow deep learning apps that run directly inside the RICOH THETA camera using the internal camera OS

Tensorflow deep learning application running inside RICOH THETA camera

Makota Shohara recently published a tensorflow-theta code repository on GitHub for a workshop he ran in Tokyo. The code runs inside the RICOH THETA and provides features such as object identification.

Note: if you want to run TensorFlow in the THETA first prior to working on the code, read Running Tensorflow on RICOH THETA V

Additional Features

Speech Recognition

Tensorflow Speech for RICOH THETA

Stylize

Tensorflow Stylize on RICOH THETA

Classify

Tensorflow Classify on RICOH THETA

Android Examples

As the RICOH THETA V runs Android internally, the demo uses the android examples from the GitHub repository in /tensorflow/examples/android/.

Machine Learning Flow

Using a machine learning model, the THETA V can infer the correct answer with Tensorflow.

THETA V model for learning to predict correct answers

RICOH THETA Tensorflow Plug-in Architecture

RICOH THETA Tensorflow Plug-in Architecture

Running TF Detect Overview of Steps

  1. Modify code to work with the RICOH THETA Camera API
  2. Import RICOH THETA pluginlibrary
  3. Detect objects and shoot picture. — Using the Plugin SDK. — Use camera API for inference -> stop inference -> Shoot with Web API -> resume inference with Camera API
  4. Fix PluginLibrary — Add functionality to upload to Google Photos

Follow the 4 Steps with GitHub Code Examples

Each of the steps has a corresponding GitHub branch.

Step 1 — Modify code to work with RICOH THETA Camera API

Step 2 — Import RICOH THETA pluginlibrary

Step 3 — Detect objects and shoot picture

Step 4 — Fix PluginLibrary

Finished plug-in

You can head straight to the code and work with the Tensorflow examples and modifications. Future articles will cover development tips for each of the steps.

Improving Equirectangular Detection

The current demo isn’t optimized for equirectangular images. Community member Fabien Benetou is looking at this problem and passed along a link to this paper.

If you have anything to add, please pass on the information.

Next Step

Step 1 — How to Set Up the THETA V for Tensorflow

Related Article

THETA pluginlibrary lifecycle

San Jose Microconference

There’s still time to sign-up for the November 10 microconference in San Jose.