Training the DashLight Object Detection model in Create ML

At Skafos, we built an iOS app called “DashLight” that locates and classifies icons on your car dashboard with the iPhone camera.

It has happened to all of us: we are driving along, and suddenly a dashboard light comes on. Instead of searching through a user manual by hand to find an icon’s explanation (tedious and time-consuming at best!), you can use DashLight to discover the identity and purpose of the illuminated icon. This iOS app demonstrates a real world use-case that can be addressed with applied machine learning at the edge.

DashLight overview.

In a separate blog post, I give an overview of object detection models and discuss the process of collecting training data for the type of model that’s powering the Dashlight application under the hood. If you are interested in exploring different ways to generate training data for object detection models, I recommend reading it! In the rest of this article I will show you the steps required to train the Dashlights object detection model with the new Create ML app, available in macOS 10.15.

Gathering Training Data

Summarizing the previously mentioned article, there are a number of different approaches to generating training data for object detection models. For Dashlight, since we didn’t already have a bunch of labeled training images of car dashboards and we couldn’t use an out-of-the-box method like Turi Create’s One Shot Object Detection, we had to build our own.

Here is a link to the Github repository with the Python code to generate your own synthetic training data given backgrounds and objects.

For the DashLight app, we generated thousands of training images like the following:

DashLight training data examples.

Surprisingly, this initial model did not perform very well! We had lots of issues getting it to properly recognize icons with the phone. So, we tried a different approach: we used a smaller set of data, collected by hand and labeled with RectLabel. Though we used a smaller dataset, the higher quality images produced a higher quality model. Soon I will release a separate blog post where I dive into that issue specifically as well as others.

Training with Create ML

Once you have a set of images for your training data, training a model with Create ML is a breeze and moves pretty quickly, especially if you have access to a GPU.

Before starting training, navigate to the CreateML app in the finder, right-click on the icon and select “Get Info”. This will open the panel shown below. Make sure that the option to prefer external GPU is selected. This option displays only if you have one connected. You’ll thank me later!

Tell Create ML to prefer an external GPU.

Now, open the Create ML application and select Object Detection from the templates menu. Fill in a Project Name, Author, License, and Description as you see fit. Then, you’ll be presented with the following screen, where you can load in your data.

As shown above, the top panel inside the interface asks you to select a training, validation, and testing dataset. Hit “Select Files…” and use the browser to select a folder that contains 2 things: annotations & a folder of images. If your annotations file and images subdirectory do not have the structure shown below, Create ML will throw an error. Note that the annotations.json file contains the bounding box coordinates for each of the objects you are trying to detect. In this case, the bounding boxes delineate the location of each lit up icon on the dashboard.

Data source structure for Create ML.

Once you’ve configured the data sources, you will need to set the max number of iterations you want the model to train for.

Set the max iterations for model training in Create ML.

If you need more control over model training, or would like to adjust parameters other than maximum iterations, I suggest using Turi Create’s Object Detection toolkit in a Python environment instead of Create ML.

Now we are ready to train the Object Detection model! Hit the Train “play button” in the top left corner of the app. For DashLight, I trained the model for ~1000 iterations, running 1.5 hours with an AMD Radeon RX Vega 64 GPU connected to my laptop.

Testing the Model

Once training has completed, you should see a nice graph of training loss over time.

According to the plot, the model improved over the first few iterations and then leveled out. Unlike other model types offered by CreateML, we don’t get to see a trace of validation loss over time, meaning the risk of overfitting is high. A model that overfits the training data has a hard time generalizing to new data in the world it hasn’t already seen. This just means that we need to be particularly thorough in our model testing. Luckily, CreateML helps us to do this.

Evaluating the Model on External Data

Create ML provides two ways to check the Object Detection model’s accuracy before deploying it.

Testing Accuracy

Back in the first step, if you selected testing data, Create ML will unleash the model on the dataset and report an accuracy metric in the top right corner. This is a great way to get a model performance assessment in bulk.

One-Off Tests

Another way Create ML can test a model is through a drag and drop interface, where you can manually drag in images and see model predictions, right in CreateML. To get to this interface, click on the CoreML artifact in the top right corner:

I selected a PNG image of a dashboard with three icons present: Emergency Brake, Tire Pressure, and Low Fuel. As you can see below, each were properly identified by the Object Detection model with high confidence!

One-off testing interface in Create ML.

While this method isn’t necessarily a good measure of model strength over large sets of data, it provides a way to peel back the onion layers of the model and explore its tendencies with specific test cases.

The Output

At the end of this training process, we now have a Core ML model that we deployed to the DashLight app with Skafos. Using these steps in CreateML, you can also build an Object Detection model. Directly from the Create ML interface, you can drag this model to your desktop, Xcode, or elsewhere, enabling you to easily take the first steps towards building an ML-powered app.

Core ML model artifact.

At Skafos, we are committed to helping you build the next generation of ML-powered apps. Sign-up today for a free account and use our dashboard to manage, deploy, and monitor machine learning models at the edge.