Activity Classification for watchOS: Part 3

7 min readApr 8, 2019

Building an Activity Classification iOS App with Skafos.ai

Update: 5/29/19

This post was written for use with our legacy Skafos platform. While much of the content will still be useful, as of 05/29/19, a new version of Skafos was released, streamlining model delivery to the edge. Sign-up for an account, join our Slack community, and explore some example models to get started.

Part 2: Model Training and Deployment

This post is the highly-awaited 3rd installment in a series devoted to activity classification leveraging machine learning and on-device sensor data. In part 1, I shared about the tedious data collection process. Then in part 2, I showed you how to put the data to good use, training an activity classification model with Apple’s Turi Create.

At this point in the journey, you have the following:

Cleaned time series data from watchOS or iOS sensors labeled with activities (moving, sitting, etc)
A trained activity classification model ready to push to the edge. Don’t have one? Here’s one you can use to start.

This last post will take you through an example iOS application (for iPhone) that leverages the machine learning model you’ve prepared and Skafos.ai for delivery.

The Caveat

Since I wrote the last post, one thing stymied the plan for the initial app idea “Sedentarian” (read about that idea here):

Using the sensors of a watchOS device in the background is more complicated than it sounds. Given that my personal skill set is around machine learning, my initial idea was rather optimistic.

But that’s okay! Between different folks requesting access to an example app, and my own stubbornness, I forged ahead anyways. This post will start you down the happy path by exploring a standard iOS application for iPhone.

Application Design

Here I will introduce the basic flow.

User opens app.
App begins streaming accelerometer and gyroscope data through the model.
Trained model classifies the current sequence of sensor readings every 3 seconds.
The UILabel updates each time an inference is made by the model.
When a new model is delivered via Skafos, the sensor data inference loop pauses, resets, and continues with the new model once loaded.

This is more elementary than the initial app idea I called “Sedentarian”. The original plan was to run this app in the background, log activity classifications, and report aggregate activity at the end of the day. In the spirit of shipping something lightweight, still demonstrating CoreML model integration in the iOS ecosystem and Skafos Asset Delivery, we trimmed down the requirements! Near the end, I will add some tips and recommendations for modifying this app to meet your use-case.

Let’s dive into some Swift code!

Application Components

If you want to follow along in the code yourself, head to this github repository and fork away. That’s probably the easiest way to proceed. Disclaimer: This example app is hot off the press. If you happen to find any bugs or issues while messing around, go ahead and open an issue on the repository and I will take a look!

App Delegate

We import the Skafos framework and initialize within the application launch delegate call. All this requires is your unique publishable key associated with a project on the Skafos.ai dashboard. I truncated the code and left out the other app delegate calls because I just left them blank for now.

import UIKit
import Skafos@UIApplicationMainclass
AppDelegate: UIResponder, UIApplicationDelegate {
    var window: UIWindow?    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
       // Skafos publishable key is required
       // Find it under "app settings" on the Skafos dashboard
       
       Skafos.initialize("<YOUR PUBLISHABLE KEY>")
       
       return true
    }

View Controller

Almost all of the magic lives in the view controller. First, we import the required frameworks: Skafos, CoreMotion, CoreML, UIKit, and SnapKit. In our UIViewController class, we initialize a bunch of attributes including:

Modeling constants and initial CoreML model
Device motion data buffer used for prediction (MLMultiArray)
Output Neural Network states to feed back into the model each time an inference is made (MLMultiArray)
CoreMotion Manager class
UILabel that will present the predicted activity on the main screen of the app

Check these out below:

// Define some ML Model constants for the recurrent network    struct ModelConstants {
    static let numOfFeatures = 6
    // Must be the same value you used while training
    static let predictionWindowSize = 30
    // Must be the same value you used while training
    static let sensorsUpdateFrequency = 1.0 / 10.0
    static let hiddenInLength = 200
    static let hiddenCellInLength = 200
}// Initialize the activity classifier model and assetName
private let classifier = ActivityClassifier()
private let assetName:String = "ActivityClassifier"// Initialize the device motion data buffer for predictions
var currentIndexInPredictionWindow = 0
let predictionWindowDataArray = try? MLMultiArray(shape: [1, ModelConstants.predictionWindowSize, ModelConstants.numOfFeatures] as [NSNumber], dataType: MLMultiArrayDataType.double)// Initialize hidden and cell output layers of the Recurrent NN
var lastHiddenOutput = try? MLMultiArray(shape: [ModelConstants.hiddenInLength as NSNumber], dataType: MLMultiArrayDataType.double)var lastHiddenCellOutput = try? MLMultiArray(shape: [ModelConstants.hiddenCellInLength as NSNumber], dataType: MLMultiArrayDataType.double)// Initialize CoreMotion Manager
let motionManager = CMMotionManager()// Initialize the label that will get updated
private lazy var label:UILabel = {
    let label           = UILabel()
    label.text          = ""
    label.font          = label.font.withSize(35)
    label.textAlignment = .center
    label.numberOfLines = 0
    label.lineBreakMode = .byWordWrapping
    label.textColor = .white
    
    self.view.addSubview(label)
    return label
}()

When the user opens the app, the viewDidLoad() method of the view controller gets called. We do the following:

Stop any device motion data activity (if there was any to begin with) and reset.
Prepare the app for Skafos.ai asset delivery. I will describe this in more detail in the next section.

Here is the override of the viewDidLoad() method:

override func viewDidLoad() {
    super.viewDidLoad()
    self.view.backgroundColor = .black
    self.title = "Activity Classifier"    // Make sure nothing is running through the model yet
    stopDeviceMotion()// Skafos load cached asset
    // If you pass in a tag, Skafos will make a network request to
    // fetch the asset with that tag
    Skafos.load(asset: assetName, tag: "latest") { (error, asset) in          
        // Log the asset in the console
        console.info(asset)
        guard error == nil, let model = asset.model else {
            console.error("Skafos load asset error: \(error?.localizedDescription ?? "No model available in the asset")")
            return
        }
    
        // Assign model to the classifier class
        self.classifier.model = model
    
        // Start running the app
        self.startDeviceMotion()
    }
    
    /***
    Listen for changes in an asset with the given name. A
    notification is triggered anytime an asset is downloaded from
    the servers. This can happen in response to a push notification       
    or when you manually call Skafos.load with a tag like above.
    ***/
    NotificationCenter.default.addObserver(self, selector:
#selector(ViewController.reloadModel(_:)), name: Skafos.Notifications.assetUpdateNotification(assetName), object: nil)
}

Embedded above are two key methods:

startDeviceMotion() — checks to see if device motion is available, begins serving device sensor data, updates a buffer of data containing X, Y, & Z coordinates for user acceleration and rotation rate, every 30 samples (ModelConstant.predictionWindowSize) a new prediction is generated and assigned to the UILabel.
stopDeviceMotion() — checks to see if device motion is available, stops all device motion updates, and resets all initial values.

Think of them like an ON / OFF switch for the application’s activity classification process. There are certainly other ways you could design this, like putting those methods behind buttons or other interactive UI components.

Skafos Delivery

An asset is just a bundle of files delivered to devices over-the-air, collected and unwrapped with the Skafos Swift framework. For our example app here, the asset contains a CoreML model for activity classification: ActivityClassifier.mlmodel.

Without having to resubmit your app to the app store, Skafos will allow you to deliver updated assets (including models or other pieces of meta data) to all of your devices.

Why might you need to update your model? Plenty of reasons!

Models degrade over time. As the world changes around us, a model that you trained will only be as accurate as the training data is representative of the problem space.
IT’S A BUG! Just because you’re a lethal engineer or data scientist doesn’t mean you are immune to delivering a bug every once and a while. It happens. With Skafos, you can easily roll-back your breaking model or push out a fix.
It sucks. We firmly believe that you need to start with a walking skeleton to get things productized faster. Otherwise, you will find yourself stuck in the classic data science loop of optimization before integration. Trust me, I’ve been there. Get an initial model integrated in your app to test first. Then update it over time as new training data becomes available. This is true for all apps, not just our activity classification example.
You want to classify new activity types. The initial model I provided only classifies activities as Moving, Standing, or Sitting. If you retrain your model to identify other activities, you could deliver it over-the-air pretty quickly.

Next Steps

Thanks for following along! I hope you’ve learned more about building an ML-powered app, from data collection to iOS app design. Stay tuned to our Skafos.ai channel on Medium for another post showing you how to port this over to a watchOS application!

Join our Slack community.
Find us on Reddit.
Expand the boiler plate activity classification example to meet your own needs.
Comment below with other ideas, findings, or ways that you’ve used machine learning to power applications at the edge!