Build an Image Classification App

A guide to building an AI model using TensorFlow, Python, matplotlib, and packaging it for the web using Ruby on Rails

Don Restarone
Oct 15 · 6 min read
woman reaching out to pat a horse on the nose, with colored rectangles drawn around the woman and the horse
woman reaching out to pat a horse on the nose, with colored rectangles drawn around the woman and the horse
Original photo by Daniel Cano on Unsplash

I was recently tasked with building a web API for fruit classification. I realized that such a product would need to have the ability to detect general objects to weed out images of non-fruits. So I decided that I would start the project by getting my hands dirty by exposing a TensorFlow model to the web.

In this article, I will cover the functionality and architecture of this app from a practical, code-first approach. At a high level, the project will be split between its AI and web layers, and I will give you a guided tour of how I glued them together.

In case you want to jump straight to the code, check out the GitHub repository. An installation guide is included in the repo so you can take the app for a spin locally (tested on Mac and Ubuntu). You can also check out the live demo here.

So without further ado, here is a general object classification model I hacked together with TensorFlow, Python, and Ruby on Rails.

Here is a live demo of me uploading an image for analysis

App Features

  1. Users should be able to upload an image and see a preview.
  2. Users should be associated with their uploaded image (but without login/authentication) so we can present the correct image back to them once it is analyzed.
  3. Image analysis should present the user with labeled bounding boxes drawn on the image they uploaded.
  4. Ensure that the resolution of the image output is fixed regardless of the original image size (scale all images to 800 x 800).
two photos, one of a group of people before the analyzer is run and one after showing rectangles around each person
two photos, one of a group of people before the analyzer is run and one after showing rectangles around each person
When the user clicks Analyze (top illustration), they should be shown a loader and once the system is done processing— it should show the analyzed image (bottom illustration)

From an architectural perspective, to implement the above, we will use Rails, ImageMagick, and some JavaScript to handle the file upload, image preview, and scaling (to 800 x 800).

To associate users with their uploads, we will use Ahoy, which will generate a unique token for each visitor. This token will be used to reference the uploaded image and its analyzed output in the file system.

Once the file is uploaded, saved, and scaled to 800 x 800, we will use some basic Bash shell calls from Ruby to pass the image to the Python script for processing. Once it's processed, we will grab it from the output directory (where it's placed by the Python script after analysis) using Ruby and present it to the user through the Rails stack.

Now that we have a gameplan in place, let’s get to work!

The Rails Side

The Rails application in my repository is stored under the object-detectr/potashdirectory. Set up Active Storage in Rails as per this tutorial, and set up the following entity relationships:

rails g model Image analyzing:boolean analysis_completed:boolean assertion:stringrails g model ImageAnalysis image:references

After generating the models, install Ahoy. Once the tables are set up, your database schema should look like this:

db/schema.rb

Jump to app/models/image.rb and drop in the following:

app/models/image.rb

This will tell Active Storage that this model has an attachable file named captured_image and it has one image_analysisassociation. We will use the captured_image attribute to store the image given by the user and the image_analysis association to store the analyzed image which, at the end, will be presented to the user.

app/models/image_analysis.rb looks identical to image.rb. (This is a code-smell for DRY, and you can unify this into one model easily..)

app/models/image_analysis.rb

With this in place, let’s generate a controller that will render a view and handle the file upload.

rails g controller Images new create

For convenience, we will change the route file to present the user with the page for image upload when they visit the root path.

config/routes.rb

Drop in the plumbing that handles the image upload, scaling, and Python script invocation. While you’re at it, feel free to call the code police because I just broke the single-responsibility principle to smithereens. I put it all in one file in anticipation of making explaining what’s going on a little easier — guilty as charged.

app/controllers/images_controller.rb (Don’t be like me, please separate your concerns.)

So let’s break it down. The new action renders the view that allows the user to attach a file for analysis (more on this a little later) and the before_action Hook ensures that the user must attach a file before moving forward.

The action happens in the create method, where we grab the unique token for the current user session from Ahoy with the call to current_visit.visitor_token. Using that token as an identifier, we create the image, scale it, and grab the path to the scaled image.

Once the paths to the scaled image and the Python script are calculated, we make system calls (lines 15–16) to copy the scaled image to the directory that expects the input (on the Python side) and run the Python script that will call the TensorFlow model.

After the Python script is run, we compute the expected file name of the output and the path to the directory where it will be placed. If the file has been created without error (line 20), we create an ImageAnalysis instance and attach the analyzed image to it.

Let’s take a look at the views rendered to the user. The new action will present the user with a form and a submit button:

app/views/images/new.html.erb

We will use a bit of JavaScript to render a preview of the uploaded image so the user will be able to see what they attached before they upload it to our server.

app/javascript/application_scripts/main.js

An onChange listener is added to the form, which fires the function readURL that sets the source attribute of the img tag to the selected image. With this code in place, we are able to render a preview as soon as the user attaches an image to the form. Let’s take a look at the view that renders the output once they click Analyze!:

We check if the processed image exists (line 8) and if it does, we render it; otherwise, we present an error, asking the user to give us another chance.

All right, that’s all for Rails/Ruby. Let’s move on to the Python side!

The Python Side

The TensorFlow models live under the object-detectr/models/research directory of the project. These models were cloned from the official TensorFlow repository.

After making sure that your Python version is 3.7.9, run the setup script and build the project:

cd object_detectr/models/research
python setup.py build
python setup.py install
python -m pip install TensorFlow==1.15 lxml pillow matplotlib jupyter contextlib2 cython tf_slim

After building the project and installing dependencies, jump to models/research/object_detection/run.py:

models/research/object_detection/run.py

I grabbed the original version of this script from this repository and adapted it to my use case. If you want an in-depth view of the script, check out this excellent video by the author of the repository mentioned above.

My changes to the script are as follows:

  1. Allow for a string to be passed as a command-line argument, and use that to point to the input file name when constructing the TEST_IMAGE_PATHS array (line 102).
  2. Modify the output (line 147) to prefix the generated image name with the string that was passed in from the command line.

These changes allow Rails/Ruby to programmatically place an image in themodels/research/object_detection/test_images/ directory using the unique identifier generated by Ahoy as the file name. Once that is done, the script can be invoked with python run.py some_unique_identifier.jpeg. Sincerun.py will include the file name that is passed in from before as the prefix to the generated file name, we can expect to have the analyzed file placed in the directory models/research/object_detection/outputs/with the name some_unique_identifier.jpeg.png.

If this file exists at the end of this somewhat unwieldy Ruby-and-Python song and dance, it will be presented to the user.

In case you made it this far, for your efforts, here is an analysis I ran on my housecat, Tiberius:

Yup, that’s definitely a cat.

Thanks for reading and keep on hacking!

References

  1. Bengemon825. “Bengemon825/TF_Object_Detection2020.” GitHub, 10 July 2020, github.com/Bengemon825/TF_Object_Detection2020/blob/master/scripts/updated_old_example.py.
  2. Clark, Mike. The Pragmatic Studio, pragmaticstudio.com/tutorials/using-active-storage-in-rails.
  3. Tech, Lazy. “How to Install TensorFlow Object Detection in 2020 (Webcam and Images!).” YouTube, YouTube, www.youtube.com/watch?v=usR2LQuxhL4.

Better Programming

Advice for programmers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store