Diving deep into fitness and data science with the SmartGym team
Author: https://medium.com/@axellau
From December 2021 to June 2022, I was fortunate enough to get an opportunity to intern with the SmartGym team under GovTech’s Sensor and Internet of Things (SIOT) division. Find more information about SmartGym here.
I joined the team in late December 2021, just a week before the Christmas break and was introduced to my role in the team, which was to work on improving the Bodyweight Station alongside my colleague Joaquin.
Improving the exercise detection capabilities of the Bodyweight Station
You might be wondering what the Bodyweight Station is. It is a stationary station powered by the Nvidia Xavier NX, which helps to track various bodyweight exercises such as push ups, sit ups and planks. The exercise detection comprises 2 steps.
- The Movenet pose estimation model estimates your body landmarks to show the position of your joints such as shoulder, elbow, hip and knees. Movenet is a lightweight pose estimation model introduced by Google for mobile devices and edge devices. Find more information about it here
- A k-nearest neighbours(k-NN) model classifies the current pose that you are in based on your current body landmarks.
My role involved working on the data-science aspect of the Bodyweight Station, such as improving the model and finding ways to optimise the performance of the models to improve the overall performance of the Bodyweight Station.
By the end of the 6-month internship with the team, I helped to improve the Bodyweight Station in terms of the following metrics:
- Improved average fps (frames per second) of the station from 22 to 30 fps. A higher fps on the Bodyweight Station allows us to better track exercises as the station is able to capture more frames of the user in the various exercise positions, which will allow the Bodyweight Station to better identify the current pose of the user.
- Re-trained a new k-NN model to help better differentiate between the various exercises and re-introduced planks to the set of exercises detected by the pose classifier model.
- During the internship, I had experimented with various camera angles for the Bodyweight Station. Ultimately we decided to change camera angles since the initial camera angle did not produce good results in terms of visible body landmarks, which affected the classification accuracy of the station.
Lessons learnt
Working with new Machine Learning Frameworks
During my internship, one of the tasks that I had worked on was the porting of the entire system to use an alternative ML framework instead of Tensorflow for running the Movenet Pose Estimation model.
The first framework that I initially experimented with was Nvidia’s TensorRT framework, which was optimised for running ML models on Nvidia hardware. Converting the model to TensorRT resulted in worse performance than when the model was running on Tensorflow as Movenet’s custom operations implemented in its Tensorflow model were not entirely supported in TensorRT.
The second framework I tried was ONNX (Open Neural Network Exchange), which also had a TensorRT runtime that could make use of the Tensor cores inside the Xavier NX to accelerate ML workflows. At the end of this process, Movenet went from running 22 fps on the Xavier NX using Tensorflow to 30 fps on the Xavier NX while running on the ONNX framework.
Fast Prototyping
The lesson that I took away from this experience was the importance of fast iterations in software development. Given that there were multiple machine learning frameworks available online, I had spent a lot of time trying to evaluate the theoretical pros and cons of each framework, similar to how project reports in school are done. Instead, what I should have done was to experiment with each of the frameworks and try to get the Movenet model running on the various frameworks. Coming up with quick prototypes would have saved much more development time and allowed us to evaluate the performance of the various ML frameworks running on the Xavier NX.
Working with Real World Systems
Prior internships that I had participated in usually gave me one-off data analysis projects, which would be completed before the end of the internship. Working with the Bodyweight Station was not a one-off task however, as there were various tasks that I had to work on to refine the performance of the Bodyweight Station such as refactoring code for better performance or improving the pose classification model. Aside from the data science aspect of the Bodyweight Station, I was exposed to how the entire stack of the Bodyweight Station worked. Gaining a deeper understanding of how the Bodyweight Station worked (Frontend + Backend) allowed me to identify where errors came from when testing my code.
Non-technical takeaways
Role of release leads
During the internship, I also had the chance to be the release lead for a single release cycle. I am really grateful for this opportunity to be a release leader as it allowed me to experience the responsibilities as a release lead such as facilitating the development cycle. Being the release lead also allowed me to gain a sense of ownership for the release cycle itself and understand the work that goes on behind the scenes to facilitate a smooth release cycle.
Importance of Team Culture
While I was onboarded onto the team remotely, I felt that I was a part of the team due to the various team bonding opportunities that I experienced. The team made it a point for us to set aside time for a monthly team lunch and weekly sharing sessions. It was during these sessions that I got to know more about my fellow team members and their hobbies outside of work.
Another point that I greatly appreciated was that the team was very warm and welcoming to new members/ interns joining the team, dedicating a lunch and an introduction session whenever a new member joined the team. This really helped to integrate new team members, which helps to make them more willing to ask for help in work when they need it.
Final thoughts
Overall, I thoroughly enjoyed my internship with the SmartGym team. It was definitely a challenging internship, with its ups and downs. I am grateful for having the opportunity to combine my interest in data science with my interest in personal fitness. If you want to experience a technically challenging internship with a warm and welcoming team, do apply for an internship with the SmartGym team!
