Machine learning on iOS became trendy when Apple introduced CoreML at WWDC 2017. It can be intimidating at first with all the new concepts, frameworks and models, so we started with cloud services to learn basic machine learning, dataset preparation and training in IBM Watson and CoreML (Part 1) and Azure Custom Vision and CoreML (Part2).
Using cloud services is convenient. However, it costs money and has certain limitation on how we want to customise our model training. And not to mention the privacy issue with all of the user’s images being uploaded to someone’s servers.
Today we try a less easy approach: using a framework to train model locally. There are many frameworks like TensorFlow, Keras, Scikit learn, Caffe, etc. We will work with Turi Create because it’s relatively easy to set up and it can export CoreML compatible model on the fly. Also, Turi Create gives us better understanding of some machine learning tasks and concepts, those will be the foundation for us to use other advanced frameworks.
In this tutorial we will build another Avengers superheroes classification using Turi Create. Let’s dive in!
Turi was initially a machine learning startup in Seattle, known for its product GraphLab that simplifies common machine learning tasks. In 2016 Apple acquired Turi and later in December 2017 made Turi Create as an open source project.
Turi Create simplifies the development of custom machine learning models. You don’t have to be a machine learning expert to add recommendations, object detection, image classification, image similarity or activity classification to your app.
As I mentioned before, the framework is simple to use, visual, flexible and exportable to CoreML compatible models. So what you trained can be used right away in iOS, macOS, tvOS and watchOS apps without any extra conversion.
Turi Create focuses on app domain, rather than model domain. It means that we don’t need to worry about constructing model architecture and connecting layers, instead we deal with high level APIs for the tasks at hands like loading images, training and evaluating. It also means that we don’t need to be machine learning experts to use it. For now the supported tasks are:
- Recommender systems
- Image classification
- Image similarity
- Object detection
- Activity classification
- Text classifier
In this tutorial, we use Image classification task to train custom model for our superheroes dataset. I have also put lots of hyperlinks to source code and reference in this post for you to explore further. Let’s take the first step now.
Given an image, the goal of an image classifier is to assign it to one of a pre-determined number of labels.
Step 1: Python
Like many other machine learning frameworks, Turi Create is written in Python and we need to write some Python code to call its APIs, so some basic understanding of Python is required. There are many tutorials online.
According to system requirements, Python 2.7, 3.5, 3.6 are supported, this is confirmed in this issue as from Turi Create 4.1. I’m using a MacBook with macOS High Sierra so I have Python 2.7 by default. Although there’s movement for Python 3, let’s use the system Python 2.7 for now.
You can check Python version and its executable by running the following commands in terminal:
If for some reasons you don’t have Python installed, you can install it here. Python comes with
pip, which is a package management system used to install and manage software packages written in Python. We need pip to install turicreate, run the following command:
pip install turicreate
Turi Create recommends using virtualenv or Anaconda to create isolated Python environment. You are free to create virtual environment, but in this post we just merely execute Python scripts for simplicity.
Step 2: Data set
We use the same data set from Machine Learning in iOS: IBM Watson and CoreML post. You can collect your own dataset or use ones in this GitHub repo. Eventually, we train with images of 4 superheroes: Ironman, Captain America, Spiderman and Thor. Images for each superhero lie in their own folder, the name of the folder can be seen as a label or a tag.
Step 3: Load data and explore
In the same directory of
dataset , create a file called
turi.python and type the following. You can use editors like PyCharm to easily edit your Python code. Type the following:
Firstly, we need to import
tc . The
import statement with
as is to define
tc as a local namespace in this file for the
In #1 As we said earlier, Turi Create has many modules for working with different kinds of tasks, they are
tc.object_detector, tc.sentence_classifier, tc.recommender, ... but here we need to use
tc.image_analysis because we’re working with image classification.
The load_images is used to loads images from a directory. JPEG and PNG images are supported. The
with_path parameter is to indicates whether a path column is added to the
with_path is set to
True, the returned
SFrame contains a
path column, which holds a path string for each image object. As of this line, the data object contains a table with many records for each image, with 2 columns: the loaded image data and a
In #2, we already have images as our features/input variable, but we need a class/label to classify. Here we use os module to get the folder name, as the name of each folder is the class name. For example, the
path might be
/Users/khoa/XcodeProject2/Avengers/dataset/captain_america/steve-roger-america.jpg, but the
hero_name should be mapped to
In #3 We save our data as
.SFrame object, which is a data structured known by Turi Create. SFrame has a tabular data structure that knows how to load from many common file formats like CSV and has many handy data manipulation methods. You can read more about it in Logical Filter section.
Turi Create is kind enough to provide with a macOS app that shows our
data object in a user-friendly interface, so in #4 all we need is to call
So go to terminal to execute our script:
After a short while with some “Unsupported image format” (Don’t worry, it’s because of some malformed images that I downloaded from Google), you will get a saved
turi.sframe folder on disk and the app Turi Create Visualization appears. Here you can see we have a table with many records and 3 columns
Step 4: Train
In this step, we will use the saved
turi.sframe to create our model. Add a file called
turi_train.python and type the following:
We first load the saved
turi.sframe into an
SFrame object called
data . Then we split the dataset into train and test data. It is a good practice to use about 80% or 90% of data for training, and the rest for validating and testing. This way we are sure that our trained model will work.
In #3 we create a classifier model based on the train data, with
hero_name as the label for our images. The model is of type
ImageClassifier , you can read more about creating function of module
image_classifier here. This model is a trained model and you can use it for prediction or classification (as known as prediction with confidence).
predict method in #4 is for demonstration purpose only. It performs prediction for our test data and returns an
SArray object with the labels for each image in test data. Something like below, you can
[‘captain_america’, ‘thor’, ‘captain_america’, ...]
evaluate method in #5 is very handy, as it evaluates the model by making predictions of target values and comparing these to actual values. The returned value is a dictionary of evaluation results where the key is the name of the evaluation metric (e.g. `accuracy`) and the value is the evaluation score. Here we usually get
0.9 , not very bad for our humble dataset.
Lastly in #6 and #7, we save the trained model as Turi compatible
turi.model for future use and export to CoreML model. We will soon use this CoreML model in our iOS app.
turi_train.python to execute our script. After a short while, you will get a
turi.model folder and also a file
But wait, the exported
.mlmodel is around 94, 2 Mb !! It is a bit heavy for an iOS app, let’s find a way to reduce the size.
Dig into the source code
Turi Create exposes high level APIs with some default parameters. In case you want to tweak parameters to your needs, read the official documentation, or better, consult the source code. Here are some links to very important files for image classification
- image_classifier.py: Class definition and utilities for the image classification toolkit. Contains methods
- image_analysis.py: Contains useful methods
resizefor images loading and customisation
- sframe.py: This module defines the SFrame class which provides the
ability to create, access and manipulate a remote scalable dataframe object. It has handy methods
exploreto load saved
.sframeand to explore SFrame data structure in a GUI app.
Normally the training process takes a lot of time, but Turi Create manages to do it fast. Take a closer look at the console output when you run
Download completed: /var/tmp/model_cache/resnet-50-symbol.json
Download completed: /var/tmp/model_cache/resnet-50–0000.params
Learning high-level concepts about data means that deep learning models take data, for instance raw pixel values of an image, and learns abstract ideas like ‘is animal’ or ‘is cat’ about that data
Conceptually, all this means is that you have a composition of simple non-linear functions, forming a complex non-linear function, which can map things as complex as raw pixel values to image category. This is what allows deep learning models to attain such amazing results.
It’s not uncommon for the task you want to solve to be related to something that has already been solved. Take, for example, the task of distinguishing cats from dogs. The famous ImageNet Challenge, for which CNN’s are the state-of-the-art, asks the trained model to categorize input into one of 1000 classes. Shouldn’t features that distinguish between categories like lions and wolves should also be useful for discriminating between cats and dogs?
Turi Create retrain the model based on its 2 pre-trained image classifiers Resnet and Squeezenet. Each differentiates in model architecture and size. Squeezenet supports nearly the same 1000 categories as Resnet, but with much fewer parameters, therefore Squeezenet is much smaller in size and takes less time to retrain. The performance and accuracy are apparently reduced a bit, but not a lot, so we will use Squeezenet for our app.
Detects the dominant objects present in an image from a set of 1000 categories such as trees, animals, food, vehicles, people, and more. With an overall footprint of only 5 MB, SqueezeNet has a similar level of accuracy as AlexNet but with 50 times fewer parameters.
def create(dataset, target, feature = None, model = ‘resnet-50’,
max_iterations=10, verbose=True, seed=None):
model : string optional Uses a pretrained model to bootstrap an image classifier
— “resnet-50” : Uses a pretrained resnet model.
— “squeezenet_v1.1” : Uses a pretrained squeezenet model. Models are downloaded from the internet if not available locally.
Once downloaded, the models are cached for future use.
turi_train.python and change the model to
squeezenet_v1.1 , and run the script again. After a while, you will get a
TuriCreate.mlmodel of just 5Mb, how beautiful is that. Take a look again at the console log for more information:
You can see here the number of train images (examples), classes (we have 4 classes for 4 superheroes) and feature columns (which is just the name of each hero). The training runs through 10 iteration, which is the default value for
max_iterations parameter in
The maximum number of allowed passes through the data. More passes over the data can result in a more accurately trained model. Consider increasing this (the default value is 10) if the training accuracy is low and the *Grad-Norm* in the display is large.
For more other features, read here for more information about changing models, using GPUs and deploying to CoreML model.
Using CoreML model in iOS app
Now let’s add the
TuriCreate.mlmodel model to our iOS project. If you click on our model, you should see some information about its base classifier, input and output. Xcode should also autogenerate a class
TuriCreate for us to call.
We could instantiate a
TuriCreateInput object with image of type
CVPixelBuffer and call
TuriCreate().prediction(input: input) . But for quick demo, let’s just use
Vision framework as it plays nicely with
CoreML and has the ability to load correct image format and do image resizing before prediction.
Build and run the app to see the model in action. Although the model is small, it predicts pretty well with high accuracy. Kudos to Turi Create 🤘
Where to go from here
Hope you learn something about image classification with Turi Create. Here are some links to learn more about Turi Create and machine learning in general:
- Image classification: Introduction to image classification task and simple demo for cats and dogs labelling.
- Machine learning essentials: learn about other concepts like clustering, graph analysis, regression, nearest neighbours.
- Machine Learning in iOS: Azure Custom Vision and CoreML (Part 2)
- Machine Learning in iOS: IBM Watson and CoreML (Part 1)
- How Turi Create is Disrupting the Machine Learning Landscape.