Activity Classification for watchOS: Part 2

Model Training and Deployment with Skafos.ai

Update: 5/29/19

This post was written for use with our legacy Skafos platform. While much of the content will still be useful, as of 05/29/19, a new version of Skafos was released, streamlining model delivery to the edge. Sign-up for an account, join our Slack community, and explore some example models to get started.

Part 1: Introduction and Data Collection

This post belongs to a 3-part series devoted to activity classification at the edge. In part 1, I introduced the field of “Human Activity Recognition” (HAR) and shared an idea for an example iOS application called “Sedentarian”. We also sauntered through the tedious and time-consuming process of cleaning tri-axial data from accelerometer and gyroscope sensors. In part 3 (coming soon!), I will describe how to build the app itself.

In this post we will walk through how to train a Turi Create activity classification model on the Skafos.ai platform. By the end, you will have some ideas for how to make sense of your prepared data, learn a little bit about time series classification, and get a chance to experiment yourself.

Making Sense of Sensor Data

Before diving head-first into modeling, I think it’s super important to understand the data you are dealing with. A model is only as useful as your ability to understand what it’s telling you…

With that in mind, I grabbed the cleaned dataset (described in part 1) and pulled out two continuous samples. One good strategy when dealing with time series data: make some plots. Python’s seaborn library makes it really easy!

Figure 1. Accelerometer data captured at 10Hz from a watchOS device while moving and then standing.

Figure 2. Accelerometer data captured at 10Hz from a watchOS device while sitting and then standing.

Sensor data shown in figures 1 and 2 represent different patterns of movement. We also have gyroscope observations that will come in handy when the model tries to discern between subtle activity changes where rotation rate is also important.

A couple other things to consider for your own data & use-case:

• Noisy Data: Are there any serious anomalies? Within a pre-determined window of observations (10–100), are there any peaks that are 2–3 standard deviations greater than the mean? Consider removal- deviations like this could indicate a faulty sensor.
• More Noisy Data: Does anything seem strange or unlikely? Do you see sections where the subject should have been “walking” but their sensor readings are all over the place? Consider removal- deviations like this could indicate a faulty sensor.
• Prediction Window: How many observations do you think the model needs to make a guess at the current activity? This depends on sampling frequency (10Hz), nuances between the different activity motions, and application use-case.

Let’s move on to discuss the modeling approach.

Modeling Approach

Time series classification boils down to a specific task: assigning activity labels to a chunk of sensor observations based on previous patterns. Turi Create uses a deep learning model with both convolution layers (CNN) + recurrent layers (RNN).

There are other ways to frame this problem too. However, some of the “simpler” methods not involving deep learning actually require a good bit of domain expertise and comfort with signal processing to engineer your own features. Therefore to keep it simple, we went with a well-established approach that Turi Create makes so easy…you can read more about here.

Figure 3. Rough NN architecture including input, convolutional layers, recurrent layers, and output (softmax) layers. Example taken from Turi Create.

Sensor data was recorded at 10Hz, meaning 10 observations per second. Because I wanted to generate a prediction every 3 seconds, I trained the model with a prediction window of 30. This means the model uses 30 observations across all sensors to make an activity classification (sitting, standing, or moving).

Following along with figure 3 above:

1. Chunks of sensor observations of length prediction window (30 in this case) are fed to the NN. The first layer, a convolutional layer, outputs a vector of encoded values representing learned spatial features relevant for classifying activity.
2. The recurrent layer learns temporal features across a sequence of the spatial vectors learned in step 1.
3. At the end, the NN outputs a probability vector (using softmax) over each of the three possible activities in our dataset: sitting, standing, or moving.

Introducing the Skafos Platform

Now that we’ve discussed our modeling approach, where does Skafos fit into the mix?

Why Skafos?

As a platform, Skafos provides two really nice things as a developer or ML engineer:

Figure 4. Simplified Skafos workflow.

Training and Development Infrastructure: Skafos readily provides “server-less” compute resources to perform ETL, data cleaning, or model training upstream from model deployment to the edge.

Automated Model Deployment to Edge: Once you’ve trained your models, Skafos gives you the ability to push models over-the-air to your devices. No need to move back and forth between different environments. You can also manage model versions and upload pre-trained models to your devices.

Model Training

As mentioned above, Skafos provides the platform to train these models. So, we’ve put together the code that goes along with this particular model example in this Github repository.

Check it out by creating a free Skafos login and navigating through the iOS Quickstart guide. When you reach the drop-down list of models, select “Activity Classifier”. Wait a few moments while your project is being created.

Figure 5. Select the “Activity Classifier” model from the drop-down list in the iOS Quickstart guide.

Once created, launch the project’s JupyterLab session. This will pull all required code into a workspace where you can play around and iterate on ideas. Below, I will walk you through the code I used to build this particular model. activity_classifier.ipynb serves as a template, and I encourage you to modify it to better fit your own use case, or write a new one entirely.

Following along the code in the notebook:

• Thanks to the heroic efforts in part 1 of this series, the example dataset comes pre-cleaned, though it’s not perfect. To help reduce bias in the model, ideally you’d train on data from many sessions and many subjects. Good news — by following the steps in part 1, you can create as much training data as you’d like!
• After splitting into training and testing sets, we use the Turi Create library to train a basic activity classifier in a matter of seconds using the approach outlined above. Read more about their model here.
• Before pushing the model to the edge, it’s best practice to get a sense of how the model will perform. I included a couple tests against hold-out data that serve as a starting point for model validation. In practice, k-fold cross-validation (CV) or leave-one-out CV are best for getting a feel for the model’s predictive power. I would not advise randomizing splits in the data though while training/testing because some of the features extracted by the NN will be temporal.

Maybe you’re interested in building something more complex, more hands on? In this tutorial, the author walks you through training a couple different Keras deep learning models. Best part — you can whole-sale swap out the pre-baked model example for this, export the Keras models to CoreML using the coremltools library, and then deploy in the same way through the Skafos framework. Have at it!

If you’ve made it here, congratulations! You have now built a model that can successfully classify sitting, standing, or moving.

Model Push to the Edge

As a quick aside, too often we machine learning engineers get to this point and things stagnate. In the world of sales there is a relevant phrase that comes to mind: “if it’s not paper, it’s vapor!”. A model on the shelf is useless and has no chance of providing business value. Skafos eliminates deployment bottlenecks by enabling the developer to save & push their models to an app¹ over-the-air.

Near the bottom of the activity_classifier.ipynb notebook, the model is exported to CoreML format and saved through the Skafos SDK:

Figure 6. Saving a trained model through the Skafos SDK.

Once saving is complete, if you check back on the dashboard, you should be able to see a list of your available models:

Figure 7. Listing available models.

For now, the most recent model is pushed out to your app². Very soon Skafos will provide devs the ability to push models by environment — production, or staging.

¹ We’re starting with iOS ecosystem and moving our way towards Android, Web APIs, and more.

² Fill out the App Settings shown in Figure 7. Don’t have an app? Stay tuned for part 3 of this blog series!

What’s Next?

While not intended to be a thorough mathematical introduction, hopefully you learned a little bit about making an activity classifier model. The next post in this series will address how to integrate this model within an iOS example app. Stay tuned!