Ped Motion

Philippe Gaudreau is an Insight alumnus from the Silicon Valley Summer 2017 Data Science Fellowship. While at Insight, Philippe developed a panoply of ML models to identify full golf swings and putts using GPS and accelerometer data. He completed this project in partnership with the company Ped Motion.

Golf is one of America’s favorite pastimes. In fact, there are 14M US golfers who play once a month on average (during the golf season). A 2009 survey of the American golfer conducted by Golf Magazine found that the average golfer spent $2,776 per year on golf-related expenses. Yet, there’s still very little technology introduced in the past years to help golfers with their play. In fact, existing technologies is based on physical devices used by the golfer or that are attached to the club, both interfering with the natural experience of play.

With recent advancements in Machine Learning and the mountains of data that can be acquired by sensors, Ped Motion sees a big opportunity to build a Golf shot tracking application that would also address some typical pain points experienced by golfers. Mainly:

  • Manual tracking of scores during a golf round.
  • Sharing information with friends and coaches about performance.
  • Receiving personalized advice to help improve performance.

The Problem

In order to provide such a product to golfers across the US and the world, Ped Motion asked for two deliverables:

  1. Build a classifier capable of detecting the type of golf swing (full, putt, none) taken at any given moment in time.
  2. Assert the GPS location of the golf swing when detected.

Data Acquisition

Ped Motion was kind enough to provide me with around 2GB of raw data from 24 rounds of golf with 491 labeled shots.

An Initial Approach

Golf swings don’t occur too often during the many hours spent on a golf course. In fact, finding when a swing occurs is much like finding a needle in a haystack. Originally looking at the time-series data, it seemed very difficult to detect obvious signs of swing activity.

The following figure displays the yaw,roll and pitch of the sensor over the span of one hour and 45 minutes.

Drawing from ideas of anomaly analysis, I used the GPS data to assess relative periods of time when the golfer was not moving. When the distance between two consecutive GPS points was larger than 15 meters, I discarded the data. This resulted in clusters of points that can be seen in the following picture.

GPS data after threshold for movement
Ground truth labels for golf swings in the same golf session.

Not only did this seem like a reasonable approach, it also proved to be an effective way to reduce the hay in the hay stack so to speak.

Using the algorithm DBSCAN, I was able to assign cluster labels for all the remaining points for any given golf session. The next task ahead was to assign a label for a given shot (full, putt, none) for any given cluster.

The reason I wanted to assign a label for an entire cluster rather than individual points came to me after inspecting the class distribution of my labels. In my current dataset, only around 2.5% of my data comprises of putts, another 2.5% for full swings and the rest comprised of no shot taken at all. With such severe class imbalance, even a model with 99% accuracy on all three classes would still yield many false positives when tested on the thousands of points present for any given session. Classifying a cluster as a whole is a reasonable way to mitigate this problem.

With my clusters labeled, it was time to build a classifier capable of differentiating between no shot taken, a full swing and a simple putt. In order to do so, I segmented the remaining data into two second intervals and engineered 200 features from these two second windows. Posted below are four examples of feature distributions over the three classes.

The provided examples display:
1) accel_futr_max: The maximum value for the acceleration of the sensor in the next 2 second window.
2) rot_futr_mean: The mean value for the rotation of the sensor in the next 2 second window.
3) rot_prev_max: The maximum value for the rotation of the sensor in the previous 2 second window.
4) rot_std: The standard deviation of the rotation of the sensor in the present 2 second window.

An Initial Model

With these features in hand, I decided to build a Random Forrest Classifier (RF). An RF model possesses several advantages. Primarily, it is possible to obtain am estimate for the generalization error using an out of bag estimate. Secondly, unlike Logistic regressions, RFs are capable of expressing non-linear interactions between features. Thirdly, they are highly scalable and usually perform quite well in practice. Even with these considerations, I also tried tuning a Logistic Regression model as well as a Gaussian Naive Bayes model. However, the RF model was the best.

Before training, I split my data into an 80% / 20% training set and held-out set. Afterwards, I used a Random Over-Sampling technique to make the distribution of classes in my dataset more equal in my training set.

Tuning my RF model using the score metric f1-weighted, I obtained the following results.

I chose to use the score function f1_weighted to try to balance the number of false positives and false negatives. As we can see from the precision-recall curve for the Random Forrest model, the model is very good at determining when no shot has occurred. Moreover, it does not seem as good at predicting full swings or putts. However, using this model to classify a whole cluster is certainly a way to remedy these shortcomings.

(Left) Variable importance plot for RF model. (Right) Precision-Recall Curve for the RF model

Application to Clusters

Decision tree for classifying each cluster given the RF model

When trying to label a particular cluster, I proceeded by applying the RF model to every point in the cluster and counting the number of points labeled none, full and putt. From these counts, I classified the cluster by using the decision tree to the left.

This handcrafted decision tree was created with the precision-recall curve in mind. Since the model achieves different rates of precision and recall for each class, majority vote is not a good strategy for classifying a whole cluster.

Given the classification of the cluster, I assigned the GPS coordinate for the swing as the GPS coordinate associated with first occurrence of the cluster label.

Results

Applying this algorithm to all golf sessions, I obtained the following precision-recall curve. As I suspected, the approach of classifying clusters definitely pays off.

Precision-Recall curve on all Golf sessions

Additionally, I also measured certain statistics with regards to the distance accuracy obtained by using the approach above.

Average Distance Error: 1.4 ± 3.3 yards
Max Distance Error: 21.2 yards
Min Distance Error: 0.0 yards

Overall, the above approach seems to perform quite well. I’m glad the clustering approach led to such results. In the figure bellow, we can see the result of the model on a given session. As is expected with this method, shots that are taken close together such as puts will be harder to catch. However, for most other shots, the model’s performance is impressive.

(Left) Prediction of golf swings (Right) True labels and positions.

Conclusions

With this project, I explored if we could predict the location and swing type of golf swings using nothing but GPS and accelerator data.

I’ve only had a brief two and a half weeks to work on this project, and there are definitely ways I could extend this methodology. For one, I think it would be beneficial to use a time series model such a HMMs when classifying an entire cluster. HMMs have proven to be very powerful at classifying time-series data. With more data, It would also be interesting to test RNN models on the entire raw data. I have never worked with them before but they have proven themselves to be powerful models for dealing with time series data of varying lengths.

Secondly, to move this project forward, I would like to test the robustness of this methodology with new data acquired by various randomized methods.

  1. Is the model learning very specific signals due to the orientation of the sensors in the training data? I think it would be beneficial to orient the sensors differently during data acquisition.
  2. Acquiring new data from players with different skill levels will definitely allow the model to learn more robust features.

Thirdly, to track the performance of the model in the future, it would definitely be beneficial to track metrics such as:

  1. Precision
  2. Recall
  3. Average Distance Error
  4. Max Distance Error

I just want to close out by thanking my contacts from Ped Motion, Kenny, Gil, Aman and Matt. I’d like to personally thank Matt for his help and feedback with the data over the last two weeks. I’ve had a great time learning new clustering techniques as well as machine learning methods like Random Forest Classifiers. Thank you Ped Motion for allowing me to work on a non-trivial interesting problem with great application for golfers all around.