Building a Classification Model for Android, iOS and Windows ML using CustomVision.ai
This session is a primer to introduce the concepts of deep learning with a specific focus on computer vision. It covers…channel9.msdn.com
In the video I introduced the concept of transfer learning. Microsoft’s customvision.ai service uses transfer learning behind the scenes to train a classifier that can be used to recognise specific objects in images. Read more about the service. A version of the slides that I have presented are available here.
Custom Vision Service identifies whether an image is of a particular object, but not where that object is within the image.
The methods Custom Vision Service uses are robust to differences, which allows you to start prototyping with small amounts of data. However, this means Custom Vision Service is not well suited to scenarios where you want to detect very subtle differences (for example, minor cracks or dents in quality assurance scenarios.)
When you are working with training an image classifier image augmentation can be used to boost the performance of the model. It achieves this by passing training images through a number of different steps including cropping, random rotation, flips and shearing. This increases the size of the training set and the recall accuracy of the model that you produce.
imgaug is a python library that helps you with augmenting images for your machine learning projects. It converts a set of input images into a new, much larger set of slightly altered images.
My colleague Azadeh Khojandi wrote an app for her talk at NDC London where she uses the Bing Image Search API to download images and build a classifier project at customvision.ai. She then tests the performance of the model by producing a confusion matrix.
I have extended Aza’s code to perform smart cropping of the images using the computer vision service and to perform image augmentation using the Open CV library Emgu.CV
I chose to create a ten class classifier for New Zealand (and some Australian) native birds (kakapo, kaka, kea, kokako, fantail (pīwakawaka), morepork (ruru), black cockatoo, cockatoo, tui, kiwi).
I increased the size of my training sets using the image augmentation techniques described above to improve the recall and performance of my model.
The real challenge in my example is the subtle differences between the kaka and the kea. After performing the smart cropping and image augmentation the recall and performance of my model improved considerably. I changed from the General Domain to the General (compact) domain so that I could export the models.
By turning the model into the General (compact) domain I was able to export the model from customvision.ai as both a TensorFlow and CoreML model.
My colleague Jim Bennett has demonstrated how you can use these exported models to create cross platform mobile apps in Xamarin for iOS and Android.
It turns out that although the classifier is trained on photos of real birds it is pretty good at identifying them from my daughters soft toys as well.
You may have seen the announcements two weeks ago about Windows ML that is coming to the next version of Windows. Windows ML allows you to run AI models locally on Windows 10 devices leveraging the capability on the device using the ONNX.ai format.
In part 2 of the 2-part Intelligent Edge series, Rosane and Wilson give an overview of what Windows ML can do for…channel9.msdn.com
Currently customvision.ai doesn’t allow you to export in ONNX format so you will need to export in CoreML then convert the model to ONNX.
To do this you will need to use the WinMLTools and python.
Alternatively you could leverage a Linux Data Science Virtual Machine on Azure.
To set this up locally:
Install Anaconda (Python 3.6 version)
Install and Activate a Python 3.6 environment
Put conda on your path (using system environment variables), then execute:
conda create --name py36 python=3.6
activate py36 (windows)
source activate py36 (mac)
Installing coremltools using pip will default to installing in Python 2.7. If you are in a Python 3.6 environment and using Windows you will need to use the latest version of coremltools from GitHub
git clone --recursive https://github.com/apple/coremltools.git
pip install -e coremltools/
Once you have the ONNX model and the labels file you can check out the automatic interface code generation tool to create model classes in your favourite language.
I have included a Jupyter Notebook of the entire solution in python in my repo for those that prefer to working in python rather than C#.
I also published a notebook at https://notebooks.azure.com/nzigel/libraries/customvisionai that you can clone and run for yourself.
You will need to update the code with your https://www.customvision.ai/ training and prediction key and a bing image search key from https://azure.microsoft.com/en-us/services/cognitive-services/bing-image-search-api/
What I did next was update the Windows Machine Learning Explorer Sample to run my newly created Bird Classifier.
I hope you enjoy experimenting with AI on the Edge.