Building a Classification Model for Android, iOS and Windows ML using CustomVision.ai

I recently recorded a primer for the AI Show with Micheleen Harris to introduce the concept of deep learning with a specific focus on computer vision.

In the video I introduced the concept of transfer learning. Microsoft’s customvision.ai service uses transfer learning behind the scenes to train a classifier that can be used to recognise specific objects in images. Read more about the service. A version of the slides that I have presented are available here.

Custom Vision Service identifies whether an image is of a particular object, but not where that object is within the image.
The methods Custom Vision Service uses are robust to differences, which allows you to start prototyping with small amounts of data. However, this means Custom Vision Service is not well suited to scenarios where you want to detect very subtle differences (for example, minor cracks or dents in quality assurance scenarios.)

When you are working with training an image classifier image augmentation can be used to boost the performance of the model. It achieves this by passing training images through a number of different steps including cropping, random rotation, flips and shearing. This increases the size of the training set and the recall accuracy of the model that you produce.

imgaug is a python library that helps you with augmenting images for your machine learning projects. It converts a set of input images into a new, much larger set of slightly altered images.

My colleague Azadeh Khojandi wrote an app for her talk at NDC London where she uses the Bing Image Search API to download images and build a classifier project at customvision.ai. She then tests the performance of the model by producing a confusion matrix.

I have extended Aza’s code to perform smart cropping of the images using the computer vision service and to perform image augmentation using the Open CV library Emgu.CV

I chose to create a ten class classifier for New Zealand (and some Australian) native birds (kakapo, kaka, kea, kokako, fantail (pīwakawaka), morepork (ruru), black cockatoo, cockatoo, tui, kiwi).

I increased the size of my training sets using the image augmentation techniques described above to improve the recall and performance of my model.

Screenshot of my Native Birds Classifier

The real challenge in my example is the subtle differences between the kaka and the kea. After performing the smart cropping and image augmentation the recall and performance of my model improved considerably. I changed from the General Domain to the General (compact) domain so that I could export the models.

99.5% precision and 93% recall
Confusion Matrix shows a few Kaka misclassified as Kea

By turning the model into the General (compact) domain I was able to export the model from customvision.ai as both a TensorFlow and CoreML model.

My colleague Jim Bennett has demonstrated how you can use these exported models to create cross platform mobile apps in Xamarin for iOS and Android.

I leveraged Jim’s code sample (ondevice model branch) to produce an android app that can detect native birds using the exported model.

Also check out CustomVision.ai TensorFlow CLI — Offline image classification using C#.

It turns out that although the classifier is trained on photos of real birds it is pretty good at identifying them from my daughters soft toys as well.

Running the Native Bird classifier offline on my Android phone from the exported Tensorflow model

You may have seen the announcements two weeks ago about Windows ML that is coming to the next version of Windows. Windows ML allows you to run AI models locally on Windows 10 devices leveraging the capability on the device using the ONNX.ai format.

There is a sample app at https://aka.ms/winmlexplorer that can run models created in ONNX. Details on setting up your environment are here.

Currently customvision.ai doesn’t allow you to export in ONNX format so you will need to export in CoreML then convert the model to ONNX.

To do this you will need to use the WinMLTools and python.

Alternatively you could leverage a Linux Data Science Virtual Machine on Azure.

To set this up locally:

Install Anaconda (Python 3.6 version)

https://docs.anaconda.com/anaconda/install/ 
 
Install and Activate a Python 3.6 environment

Put conda on your path (using system environment variables), then execute:

conda create --name py36 python=3.6
activate py36 (windows)
source activate py36 (mac)

Install CoreMLTools

Installing coremltools using pip will default to installing in Python 2.7. If you are in a Python 3.6 environment and using Windows you will need to use the latest version of coremltools from GitHub

mkdir coremltools
git clone --recursive https://github.com/apple/coremltools.git
pip install -e coremltools/

Install WinMLTools

Convert your CoreML Model to ONNX

Once you have the ONNX model and the labels file you can check out the automatic interface code generation tool to create model classes in your favourite language.

I have included a Jupyter Notebook of the entire solution in python in my repo for those that prefer to working in python rather than C#.

I also published a notebook at https://notebooks.azure.com/nzigel/libraries/customvisionai that you can clone and run for yourself.

You will need to update the code with your https://www.customvision.ai/ training and prediction key and a bing image search key from https://azure.microsoft.com/en-us/services/cognitive-services/bing-image-search-api/

normalized confusion matrix created inside the jupyter notebook

What I did next was update the Windows Machine Learning Explorer Sample to run my newly created Bird Classifier.

Running my ONNX Native Bird Classifier in the Win ML Explorer Sample App — performance is low due to screen recording
I also had some fun seeing if I can make myself look like a bird. The best I could do was 12% kiwi :)

I hope you enjoy experimenting with AI on the Edge.