Counting exercise repetition with deep learning
Introduction
I love exercise and improve my whole body strength. I do weightlifting and body weight exercise weekly.
Volume help your body to grow and increase your strength gradually. Volume = Weight + Repetition.
From time to time you might lost how many repetition you have done while focusing on performing a challenging exercise. At least it happen to me sometime. Therefore I start to have an idea long ago on how to use artificial intelligent to assist physical exercise? Using deep learning to count how many repetition you just perform for an exercise.
I was struggle in the beginning on how to solve this problem. After a long research and reading articles, I found a viable but not perfect solution. And here are some of my progress.
Problems & Details
There are two main problems I need to solved.
The first immediate problem is how to track human joints from video?
Most exercises can be recognized by two factors, one is joint to joint distance and another is joint rotation. For example exercise push ups can be recognized by rotation of elbow joint and shoulder to wrist distance. This is why tracking human joints in real-time is important.
Second problem is how do I translate those tracked joints’ movement into a recognizable pattern?
The answer is distance and rotation of joints during eccentric and concentric exercise movement. Take a look at image below about eccentric and concentric movement for squat exercise.
From the image I can see some information below about squat exercise.
- Eccentric: From standing position to position where you have knee bending 90 degree and hip almost touch your ankle. I also call this bottom position.
- Concentric: From bottom position back to standing position.
Performing both eccentric and concentric movement number of time to achieve your desired repetition.
What I need to pay attention here are distance between hip to ankle and rotation of knee because this movement form a noticeable pattern in each movement cycle.
Methods for counting repetition
From this article there are number of methods I can use to solve this problem. However each one of them comes with pro and con.
- RepNet
- Rule-based using Post Estimation
- Signal processing
- GymCam
- Optical Flow
Although each method seems interesting, I choose signal processing method over others due to its simplicity and efficiency. Beside it is accuated enough. As article had point out some cons, non-generic, sensitive to background noise, need the zero-crossing line from reference video.
Generate exercise statistics
See the red line on the graph at right window? That is the mean for this push ups exercise. I can use this mean for counting repetition. Following are steps to generate mean from an input video.
Tracking human joints
MoveNet is a deep learning model that can track 17 key points for human body. It is accuracy and fast, in addition it is small in model size. All I need to do is to capture each frames from video and feed them into MoveNet and then the model will produce 17 key points for each frames.
Define metric for exercise
Signal processing is specific to the exercise and non-generic therefore I need to define what metrics a particular exercise need. Metric here means distance between joints or rotation of a joint. Take squat for example I would define metrics below.
- lshl-lhip-dist: left shoulder to left hip distance
- rshl-rhip-dist: right shoulder to right hip distance
- lhip-lakl-dist: left hip to left ankle distance
- rhip-rakl-dist: right hip to right ankle distance
- lhip-angle: left hip angle
- rhip-angle: right hip angle
- lknee-angle: left knee angle
- rknee-angle: right knee angle
Key points I get from MoveNet I then calculate these metrics. Aware of camera angle before calculation. Since I have no idea what angle camera will facing, the camera could be face at front or side. Therefore I have to make assumption. For squat I would assume camera is on right/left side then I would make calculation on y axis. For instance knee rotation can only be measure from left/right side not at front.
Lowpass filter
Lowpass filter is a signal processing technique and is used to filter out high frequency signals.
I treat each metrics as individual signal and they are not always come out as smooth as I expected. A example below is an image of high frequency signals and signals after lowpass filter.
Metrics are similar to the purple signal. I can see a lot of jittering. The signal after lowpass filter is the one in blue.
I need to do lowpass filter for each metrics to get rid of jittering.
The simplest way to do lowpass filter is using following formula.
- yn: Output signal
- xn: Current input signal
- xn-1: Previous input signal
- a: Coefficient value (Alpha) where the range is between 0 ~ 1. If Alpha is 1 then output signal is equal to input signal,
Remove stationary metrics
Some metrics are representative and some are not. By representative I mean frequency of signals in a metric are variant(non-stationary) whereas non-representative means frequency of signals in a metric are non-variant(stationary).
Variant signal.
None-variant signal.
The way to determine whether to keep a metric or not is to use its standard deviation. I can set a desired threshold for standard deviation and calculate standard deviation for a metric then I compare them together to decide I should keep this metric or not.
Aggregate metrics & save statistics
After all non-stationary metrics are selected then I can add them together to become one sequential signal.
Now I can save non-stationary metrics’ name and value of mean in a json file. Of course other information can be saved for further use.
As above image you can see json contain 2 exercises and there is motion_names contain names of non-stationary metrics. And some statistics under reference.
Real-Time repetition counting
For real-time repetition counting, we are still doing the same steps that I mention above and repetition counting at final steps. The flow look like below.
Get 17 key points -> Calculate metrics -> Lowpass filter -> Remove all stationary metrics -> Aggregate non-stationary metrics -> Repetition counting.
Only two steps I need to pay attention to Remove all stationary metrics and Repetition counting.
Remove all stationary metrics
I don’t need to calculate it again since metrics’ name are already saved in json file. I could simply get these metric names and remove all metrics that I don’t needed for a particular exercise.
Repetition counting
First we retrieve value mean from json file for a certain exercise.
Now how do I know a repetition had been performed? Take a look at image below.
There are 2 colours green and red in the image and they represent down state(green) and up state(red). These 3 dots are data points in the entire signal as well.
To count repetition I can just observe the pattern. If last data point is above mean and current data point is below mean then we say there is 1 repetition completed. Some problem with this method.
- Need a proper sliding window size in order to detect repetition.
- Unable to find total repetition from the entire signal.
To solve these two problems, I had done some research about pattern recognition such as Matrix Profile. I end up a simple method by myself.
Why don’t I convert signal into 0 and 1. Make any data point that is below mean as 0 and above as 1. I will call this converted signal as binary signal. From above signal image we could end up with
[1100011100] to the first greed dot. Now make sequential 1 and 0 to become 1 digit only and we have
[1010]. Then I can define a pattern sliding window look like this [010]. Finally slide window through entire binary signal. For this example only 1 pattern have been found.
- Use combination of 0 and 1 to define sliding window size.
- Able to find total repetition from signal.
- Signal length had been reduced.
Conclusion
Exercise repetition counting with signal processing can be helpful for people want to tracking their repetition during exercise. This method leverage deep learning and signal processing to achieve repetition counting and it is fairly accurate. Furthermore it can still perform counting while partial human body is occluded.
On production level it is fast and efficient enough. Of course it depend on the spec of machine it run on. Below is application integrated with exercise repetition counting.
Application download
I had made an application for this exercise repetition counting. The application is capable of repetition counting from video clip source and webcam.
If you would like to try it out here are links to download.
Source Code
All source code is hosted on Github here.
