Our Python/Flask Cat Checker works like a charm

AI for Cats vs the rest of the world

Use Azure’s Computer Vision Image Analysis service to detect cats (or any other remarkable beings)

Do you need to automatically inspect images to check if their content is suitable for your app or site? Ok: you could design your own AI system that crunches each image and spots the features you are looking for; then, you could train/fine-tune the system and deploy it into your app.
Or you could adopt one of the available AI systems ready to be used.

This article shows how to use Azure’s Computer Vision Image Analysis service to decide whether an image is about cats — or not. Guess what: you may quickly adapt our code to classify any other subject.

We made a Cat Checker service that validates a set of uploaded photos: if the picture depicts something about cats, the photo is ok — otherwise, the image is marked as invalid. The Cat Checker is written in Python and packed in a Flask application (and website).

You can get the complete code of the Cat Checker Flask application in this public GitHub repo. If you like our code, please follow us on Medium.com: you will hear from us soon.

1. What the Cat Checker does

The Cat Checker has to validate all (not yet validated) images stored in a cloud repository [to be precise, in two AWS S3 buckets: one for the full images, the other for their thumbnails].

To know which images are to be processed, the Cat Checker connects to a database with a table called tbl_image, containing the data of the target images. Figure 1 shows the table with some mock content and the respective images in the cloud repositories.

Fig.1 — The tbl_image table and the images in the cloud repositories

Each image (along with its thumbnail) is represented in a row on the table: it has its unique id, a name, the filename in the repository, the filename of its thumbnail, and a bunch of other properties. The img_is_valid column value is set to true if the Cat Checker algorithm decides that the image has some cat content; otherwise, the value is set to false. The raws with img_is_valid set to NULL are those related to the pictures that are not checked yet and need to be validated.

To sum up, these are the steps that the Cat Checker needs to do:

1. get from the database the list of all non-validated images, obtaining the cloud repository URLs of each image (and its thumbnail);
2. for each image of step 1, analyse the image (and its thumbnail) to check if it has cats in it; if this is the case, the image is valid — otherwise, it’s not;
3. update the database, saving the validity for each image in tbl_image;
4. log the operations for future analysis.

Step 2 is where Azure’s Computer Vision Image Analysis plays its crucial role. But first things first: the Cat Checker is a Flask application, and as such, it must have a way of being called via HTTP. Here is the call’s endpoint:

@bp.route('/checkcatimages-api')
def checkcatimagesapi():
   
   db_validate_images()
   return {'data':'OK', 'error':0}

This snippet clearly shows that the db_calidate_images() does all the work. Here is how it is defined:

def db_validate_images():
   #1. GET ALL IMAGES TO BE VALIDATED
   .... (code to get the images from the database)
   #2. CYCLE ON ALL IMAGES AND INSPECT THEM
   #DATA ARRAY
   image_validity = []
   for image in images_to_validate:
      #passing db for logging purposes
      if db_image_is_cat(image, db):
          image_validity.append("("+str(image['img_id'])+", true)")
      else:
          image_validity.append("("+str(image['img_id'])+", false)")
    #if there is something to update...
    #3. UPDATE tbl_image with the outcomes of the image analysis
    ..... (code to update the database)

We stripped down the function’s code to focus on the essential parts (please refer to the complete code in the project’s GitHub repo). The code follows exactly the steps outlined above: it gets all the images that need to be processed and calls db_image_is_cat() for each one. If the latter function returns true, the image is valid, and we keep track of this result by appending a “positive” SQL string in our image_validity list. Otherwise, the image does not contain feline content, and an “invalid” string is appended to image_validity.
Eventually, the database is updated with the analysis outcomes for all images.

So db_image_is_cat() is where the magic happens, as will be explained in the next section.

2. How the Cat Checker works

The code for the db_image_is_cat() is given below; again, we stripped out some of the boring parts to highlight what’s essential.

The first thing to note is that Azure’s AI services require some authentication data to work: in section 4, you will find how to get these from Azure’s portal.

This said, let’s go through the code line by line.

• In line 17, the computervision_client object is created; this will be used to analyze the images with Azure’s AI;
• In line 22 and 23, the URLs of the image and the thumbnail is composed; these URLs point to the images in the S3 buckets and are used in lines 26 and 28 to inspect the image contents via Azure’s computervision_client.tag_image() API.
• The result of the computervision_client.tag_image() function is a pretty complex object, e.g. tag_results_full_image; what is essential for our purposes is that this complex object exposes the list of tag objects that describe the contents of the analyzed image.

The Cat Checker analyses both the full image and its thumbnail: in this way, it is easier for the algorithm to spot a cat somewhere (in the full image or its thumbnail). This is why line 33 makes one unique list putting together the tag objects found in the full image with the ones found in the thumbnail.

• In lines 35 to 39 the code cycles on all tag objects that describe the image (and its thumbnail), checking if the tag’s name “has_to_do_with_cats” with a high enough confidence.

Here is the definition of the has_to_do_with_cats() and its auxiliary functions.

The reason we need to split the tag strings into tokens is that they are often composed of two or more words: e.g. “small to medium-sized cats”, “cat’s furniture”, “domestic cat”, and so on. We can then check each token with a list of cat_words that can be fine-tuned by looking at the contents of the logs.

And that’s it: this was the last trick to make the Cat Checker algorithm classify the users’ pictures!

3. Using the Cat Checker in an actual application

The Cat Checker is an essential component of our demo “CatLoader” application project.

The “CatLoader” app lets its users upload images taken from the camera or the file system (go here for a demo; we will write about it soon). The app needs to validate each uploaded image to check if it portrays a feline subject; the validation task is delegated to the Cat Checker service.

Fig . 2— CatLoader and Cat Checker working together

Figure 2 shows how the Cat Checker is connected to the “CatLoader” Flask API and HTML application. The “CatLoader” calls the Cat Checker service using a “Fire & forget” pattern: every time the “CatLoader” API needs to validate the uploaded images, it calls the non-blocking upl_check_cats_async(). Here is how this function is defined:

#CALLING CATCHECKER: request
def request_task(url):
   requests.get(url) #External URL
#CALLING CATCHECKER "async": CREATE THREAD
def upl_check_cats_async():
   threading.Thread(target=request_task, args=
                   (os.environ["CATCHECKER_URL"],)).start()

The request_task is called, receiving the URL of the Cat Checker service as a parameter.

Read the article below to learn more about the “Fire & forget” pattern.

Let Flask apps fly with “fire & forget.”

4. What is needed to make the Cat Checker run

The Cat Checker uses Azure’s Computer Vision Image Analysis service; therefore, to run our demo code, you need to do the following:

1. If you don’t already have one, create a Microsoft Azure account. Azure will require you to set up a billing profile, called Azure Subscription, to be selected later when creating the “Computer vision” service instance — and a Resource Group. You can learn more about these aspects here.
2. With the Azure portal, you can create a new “Computer vision” service: from the home page of the portal, choose “Computer Service|Computer Vision” and press the “+ Create” tab button. Fill in the “Basics” section of the creation wizard (and skip all other subsections): you should insert your subscription data and choose a region and a name for your AI instance. You may select the FREE pricing tier to start testing the service.
   Then press the “Review + Create” button.
3. Once you have created your personal Azure Computer vision service, select it in the Azure portal and go to its “Key and Endpoint” section. There you will find the Keys and Endpoint data that are NEEDED to call Azure’s Computer Vision Python functions from our/your app.

Fig. 3— Azure’s Computer vision service KEYS and Endpoint screen

Note that Azure’s client library:

azure-cognitiveservices-vision-computervision

is a requirement for any Python application that wants to use the “Computer vision” service; you will find the library and its dependencies in our requirements.txt file.

Another requirement for the Cat Checker Flask application is the database (a PostgreSQL instance) which holds the table(s) used by the Cat Checker. The /database scripts/ folder of the Cat Checker project contains the scripts to recreate the database structure that the application is expecting (read the README.md file in the same folder for instructions).

To install the Cat Checker locally or to deploy it on PaaS platforms like Heroku, AWS or Azure, you may find helpful our article “How to deploy Python/Flask on Heroku, Azure or AWS”:

How to deploy Python/Flask on Heroku, Azure or AWS

Furthermore, in the article below, you can get further info on how the Cat Checker project is structured — and how to install its dependencies and database to run it locally. In fact, the Cat Checker project was built starting from the “Minimal+DB” starter project described in the “Flask + Bootstrap 5 starter websites”

Flask + Bootstrap 5 starter websites

5. Concluding remarks

The Cat Checker works quite well: Azure’s Computer Vision Image Analysis service to validate users’ images is an excellent temporary solution.

We are working on a custom machine-learning system that has a better performance. For example, our current algorithm misclassifies some photos where the cats are portrayed “upside down”.

We are aware that the Cat Checker code can be improved and optimized. If you have problems with our project, don’t hesitate to contact us.