How to create an image recognition web-service — Part I

IMPORTANT NOTES: This tutorial was based on Tensorflow 1.0.1. As of this update, Tensorflow is on 1.7. Tensorflow hub is now the official library dedicated for transfer learning tasks.

In this tutorial, I will show us how to build a web-service for an image classification agent.

You would learn little things about:

Tensorflow, Supervised learning, image classification task, Google’s Inception-v3 model and how to build simple flask web service. I will try to avoid the technical and maths jargon that took me some pain to understand.

So why image recognition? Short answer; it’s changing business for good. Facebook, Twitter, Google, gaming companies, manufacturing companies are using one form of it or the other. There are also startups in the business of providing APIs for this task — that is how serious it gets.

For instance, at Oncenout.com( an online community for pre-owned fashion in Nigeria) where a member has to fill a form to submit images of her fashion items, we are using this technique (feature still in beta) to better the user experience by auto-tagging images with the category (and soon the colour). Submitting items now feels like posting a photo on Facebook. Can we also auto-tag the condition of the fashion item?

Once”n”Out

One hard way of going about building an image recognizer is to attempt building a model from scratch. Doing this will probably get you very grounded in computer vision, but it will take a longer time and huge computational resources to get a good result.

Now what is a model here? Think of a model as the m and c in the straight line equation: y = mx + c. If you know the values of m and c, you can predict(classify) y given x and vice versa.

One very easy approach is to use Google’s Inception-v3 model to re-train on a smaller data set. This process is technically called Transfer Learning.

In this tutorial, we will use this approach to build a web service for classifying fashion images. Our classify should be able to classify fashion items from 3 categories — Clothes, Shoes, bags. Practically, this can even be more than 100 categories. One other good category to add is ‘Unknowns’ — because your user might be uploading a refrigerator — I have seen that happen twice!

You can use a similar approach to train any image classifier tasks — you just have to have a different data set.

Step 1. Data collection
We need to get images for the re-training task.
So where can we get enough fashion images? Exactly — online fashion stores. And most fashion sites have product categories. That solves the next step’s challenge. I used this tool to scrape over 500 images from different fashion sites — names with held — of course you can use Google’s image search to trace the sources. I’m not worried.

2. Data labeling
We didn’t have to do a lot of work here because of the way the images are scraped.
I scraped form e-commerce sites that already have categories sorted.
If you don’t have organized images, you have to get some friends, buy them popcorn and beg them to help you sort the image. Or If you have a budget, you can use Amazon Mechanical Turk.

Images folder — note how they are named accordingly

3. Install Tensorflow from source on an EC2 instance.
An instance that has GPU support would save you some time.
Also, I tried installing tensorflow on EC2. I faced an error with getting CUDA installed. So I decided to use a community provided AMI on AWS marketplace. This AMI has everything pre-installed including tensorflow and CUDA. Thanks to ritchieng. I will suggest you do the same.
I choose to use g2.2xlarge EC2 instance. Please do the same.

3. Upload training set to an S3 bucket
Since we are going to be working on an EC2 instance, it makes sense to have our images on S3.
i. Zip the folder containing the folders of the images
ii. create an S3 bucket and upload the zipped file.
ii. make the bucket public to enable you to access it from anywhere.

Make sure you give the bucket the right permissions.

4. Download the zipped folder into our EC2 instance.
Next, we need to SSH into the EC2 instance.
Say our pem file is located at $HOME/pemfile.pem

$ cd ~
$ ssh -i “pemfile.pem” ubuntu@ec2–54–183–221–104.us-west-1.compute.amazonaws.com

Remember to change ‘root’ to ‘ubuntu’. This brings us to our EC2 terminal.
You should find a tensorflow folder sitting in the home directory

Using the curl command to download the training data from S3. Also we have to unzip the file.

$ wget https://s3.amazonaws.com/onomachinelearning/fashion_items.tar.gz 
$ tar xzf fashion_items.tar.gz

4. Now we need to build tensorflow from source

$ cd tensorflow
$ bazel build tensorflow/examples/image_retraining:retrain

5. Retrain the model using the inception model

$ bazel-bin/tensorflow/examples/image_retraining/retrain — image_dir ~/fashion_items

This will take awhile. You can use this time to finish up GOT.

6. Test if this works

So let us see how good our retrained model is. We are going to be using one of the images in our training data. That’s like cheating. But it tells us if our model is mentally ok to classify images it just saw.

$ bazel build tensorflow/examples/label_image:label_image && bazel-bin/tensorflow/examples/label_image/label_image — graph=/tmp/output_graph.pb — labels=/tmp/output_labels.txt — output_layer=final_result — image=https://oncenoutapp.s3.amazonaws.com/uploads/7f4c62be-614d-4f66-a445-57138f100603/_MG_4683.jpg

Typical result should look like this:

Wow! Our classifier is 99% sure that the image of the shoe is actually a shoe.

7. Move the retrained graph(model) to a permanent folder
By default, tensorflow would save the model output at /tmp/
This is a temporary folder. Contents in this folder can be wiped out if the EC2 instance reboots. So lets keep is somewhere safer.

$ mkrdir mymodel
$ cd ~
$ mkdir mymodel
$ cd /tmp 
$ cp {output_graph.pb, output_label.txt} ~/mymodel

8. At this point, we have the files needed to run predictions for a any given fashion item.
But this system currently resides in our EC2 instance. We need to provide a simple interface for calling this function and retrieving classifications.
For this, we will use Flask. Flask is the simplest python framework I know.
You can setup and run a flask application in 7mins.
So let us build a web API and an interface for calling our classifier function. This API would be pretty simple. No authentication etc.

To install flask and create a flask application, follow this simple tutorial

9. Creating a web endpoint for posting images and retrieving results

Since we have our flask app running, the next thing is to write a simple API end for calling our function.

Our main route is /classify. We have not added any view template for this endpoint yet. So for testing purposes we can only use a HTTP tool like POSTMAN to run a test. Note that we have to submit our image as binary data.

Running POSTMAN POST request and the results.

From the screenshot above you can see that our classifier is doing very well at recognizing fashion items.

So that’s it. In the Part II of this post, we will build a minimal web template for our recognizer. This will enable us use HTTP form post to the API.