The Bike, the Cam, and One Office Cyclist
We never miss a chance to do new exciting research, so when we learned there were several amateur cyclists in our team we thought there just had to be a way we could combine the two. That’s how this sort-of-scientific experiment for a potential out-of-the-lab use-case was born. But before things got too complicated, we should explain some theory behind it.
Recently we published an article where we described the existing contact and contactless heart rate tracking methods from ECG to machine learning.
In a whole lot of industries there is a growing demand for contactless heart rate (HR) measurement tools. In many cases, use of traditional methods such as EEG or ballistocardiography is impossible or inconvenient. For instance, it may be problematic to use heart rate and respiratory monitors if patients are newborns or suffer from skin conditions, or if they just move too much during the procedure.
In some cases, a finely working technology can show erroneous results if applied in complicated or just imperfect conditions.
A good example is fitness. Last year we wrote an article where we explained that most fitness rate trackers are based on the principle of photoplethysmography (PPG): they are equipped with a light-emitting diode that casts light upon your wrist, and a light sensor that registers the reflected signal. Some trackers such as chest straps essentially act as ECG devices — they measure the bio-potential generated by electrical signals that control the expansion and contraction of your heart chambers.
Regardless of the principle used, there are some issues related to wearable heart rate monitors that show during exercising. For instance, chest or wrist straps may get loose due to movement. Clothing may also become an obstacle: synthetic training shirts produce additional static electricity buildup that can mess with your tracker readings.
Respiration rate (RR) monitoring is another issue, mostly because at this point, you can’t really monitor your RR during workouts. Wrist straps are simply not equipped with such technology, and even the most expensive and sophisticated ECG devices fail when you move (which is sort of the whole point of working out).
As we said, in our team we have guys who love cycling, and while our work is not ordinarily focused on developing our state-of-the-art technologies in the direction of fitness industry, the idea of combining artificial neural networks with bicycles sounded fun enough to make an experiment out of it.
Putting it scientifically, we decided to compare the precision of our contactless HR analysis with that of contact PPG and ECG devices.
We decided to test our basic algorithm that has great potential for applications in such industries as fitness and telemedicine, and can be used to monitor physical state of people involved in high-risk jobs. But here we should note that any technology based on video feed is prone to artefacts caused by movement and changes in lighting. Thus, if we could help to tackle some challenges in the fitness industry with the early versions of our contactless monitoring tools, then this experiment would be only the first but great step on a long way of adopting our tools for it.
So we brought a bike into our office, got out some PPG and ECG sensors, and made our Product manager Nik Blinkov ride it early in the morning while observing the vast landscape of a cemetery stretching right in front of our office. (Sport is life, you know.)
Firstly, we taped the webcam to the bike handlebar stem (see Pic. 1, 2).
Video feed from the webcam was received and recorded by a computer placed in the lab (Pic. 3).
Secondly, we attached pulse sensors (PPG) to Nik’s ring finger and instructed him to move said finger as little as possible to avoid artefact creation. Then we put the ECG shimmer on him; its data was also received by the lab computer (Pic. 4, 5).
(We synchronously recorded the video feed coming from the webcam, and the ECG and PPG data coming from the sensors. See Pic. 6)
Thirdly, Nik was then asked to ride the bike for three five-minute rounds taking short breaks between rounds, 15 minutes in total. He was also instructed to increase his cadence with every new round gradually raising his heart rate (see Vid. 1, Pic. 7, 8).
After he finished, we analyzed the pre-recorded data received from the webcam and Shimmer PPG and ECG sensors; the video feed was run through our artificial neural network. Afterwards, we compared the results (see Vid. 2, Pic. 9).
The results obtained from the webcam were more accurate than those obtained from the PPG sensor as the PPG sensor could not correctly record heart rate data from a moving test subject (see Pic. 10, 11). For the same reason, the data from the Shimmer ECG sensor contained artefacts that made it partially unreadable (see Pic. 12).
In this experiment, our contactless HR measurement technology worked very well in comparison with professional wearable Shimmer PPG and ECG monitors whose estimations were a baseline for our analysis.
We see tremendous growth potential in it, especially considering the fact that such results were obtained using the basic version of the algorithm made specifically for lab environment and trained on unmoving test subjects in perfect light conditions.
However, the work of our algorithm was not compared to other professional fitness ECG trackers — which would be a great thing to do — so we are open to collaborations with sports equipment manufacturers to calibrate and refine our contactless tools for their tasks (please reach us at info@neurodatalab.com or contact@neurodatalab.com for more info).
Overall, we had a lot of fun during this experiment which involved a lot of people from our team and became sort of a team building exercise for us. We got valuable knowledge of how to match our tech with real-world use cases and how to improve it. Now, we are refining the algorithm using the results of this bicycle experiment.
Our algorithm is easily packed into iOS and Android SDK and does not require high-end video to measure HR so it could work with fitness apps. It can also be integrated into fitness equipment software to give fitness industry one more option to build seamless experiences with almost real-time video stream analysis. In addition, we train our algorithms to track respiration rate and are analyzing how our contactless RR tracker worked on this video feed. Stay tuned for updates!
P.S. You are very welcome to share your thoughts on this experiment and possible applications of our technology in your environment to get extraordinary results. Please email contact@neurodatalab.com to have a chat with us!
Authors: Marina Churikova, Research Scientist at Neurodata Lab; Nik Blinkov, Product Manager at Neurodata Lab; Francesca Del Giudice, PR Specialist at Neurodata Lab.
We also express our deep gratitude to Maria Malygina, Research Scientist at Neurodata Lab, who generously provided all of the equipment and assisted with this experiment.
