Every Beat Counts. Comparing Remote Webcam Heart Rate Detector to Wearables
First, a spoiler. Remote heart rate detection technology is as good as the coolest wearables. Good news, you can check it out for yourself on our demo portal.
Now the article. Ever since technology has become portable, it has been placed in service to manage our personal well-being. The abundance of all sorts of wearables and fitness trackers on the market signals their popularity with the people and the idea of staying in check in terms of health. Only in the US, the number of wearable device users has grown 3.5 times since 2014, up to 86 million people, making a quarter of the US population own a fitness band.
Wearable technology has been a #1 fitness trend since 2016 — and probably will stay there for some time ahead. Meanwhile, the scientific community is on track to the future and offers new methods that allow to remotely detect one’s pulse without the need to actually wear any gadgets. No gadgets at all. The inevitable fast spread of such technology would mean switching the focus from providing one person with the tools for advanced monitoring of one’s health to modernizing the whole environment around people.
We at Neurodata Lab, our R&D people to be exact, came up with a technology that is capable to track the heart rate and pulse variations via a simple webcam. That means just by turning a face to the camera. To check how accurate such technology can be we decided to conduct an experiment and compare our tech to the most popular tracking devices that set the standard in the industry, and see if remote heart rate detection software can be as accurate as an actual sensor located on the body. For that purpose we took Honor band 4, Amazfit Bip, Xiaomi mi band 3, Apple Watch Series 2, Garmin, and Samsung Gear S3.
How do wearables measure heart rate?
Heart rate variability may reflect the state of health, signal the person’s being nervous, or even become an indicator in emotion recognition, adding the ability to measure stress levels and arousal. Today though heart rate detection in wearables is mostly used for sports activities, and since recently to monitor and prevent epileptic seizures and report unusual physiological data, for instance indicating arrhythmia. The last series of Apple Watch already saved several lives by sending alerts to the users and even emergency.
In general, we can elicit two types of fitness trackers that are widely recognized: based on electrical or optical technology. The first is usually used in wristband trackers, while the second is employed on a more professional level and used in gadgets located on the chest. (Actually, Apple Watch recently got both.)
An electrical sensor measures the small electrical signals the body creates to make the heart constrict. We won’t get into technical details in this article, but the American Heart Association gives a short explanation of what an electrocardiogram (ECG) is. Portable trackers make such measurements by using electrodes that conduct a tiny bit of electrical energy to each other and then evaluate the results.
Wristband optical trackers work on a different principle. Of course, all brands aim to differ in a way in order to be more accurate and useful. The companies come up with various algorithms trying to outsmart each other, but when it comes to heart rate measurement the system is pretty universal.
Wristband optical trackers are endowed with optical sensors that can measure the heart rate with the help of light. This process is called photoplethysmography (PPG) — a light-emitting diode is shone on the skin and the sensor analyses the fluctuations in the light level which the skin has absorbed — that is how much light has returned to the sensor. The amount of reflected light changes according to capillary dilation and constriction and correlates with the heart rate.
How does remote heart rate tracking technology work?
Remote heart rate tracking works on a similar photoplethysmographic principle as optical sensors. But how exactly a webcam knows what happens inside our bodies without any close contact? The camera is located at least half a meter away from us, unlike a wristband.
The basic approach to remote tracking is to combine face recognition, detection of pixel color and skin reflectivity techniques.
First, an algorithm has to find a body area uncovered with clothes, which is usually the face and the neck. It then detects the regions of interest (ROI) where pixel colors (RGB) are measured. The relative color changes in these regions are analyzed and, in result, allow to measure the ‘pulse wave’ velocity, a consistent change in blood pressure executed on artery walls. This information allows for obtaining mentioned physiological parameters using specifically developed techniques.
To assess their quality, the results are compared with the data simultaneously collected from different wearable sensors — reference or ‘ground’ truth. Optical PPG sensors can be used, or professional medical equipment to measure ECG as well.
Actually, remote detection systems can potentially extract a wider range of physiological parameters, including respiration rate and depth, blood pressure, changes in body temperature and many more. We at Neurodata Lab are working hard to make that happen, but now will (finally) talk about the comparison experiment we carried out.
How does our algorithm compare to wearables?
An important note: there have been applications for measuring physiological signals, but they mostly used expensive HQ equipment and worked with uncompressed video. This posed a number of challenges to apply such technology outside laboratories, mainly due to a large amount of storage space and high speed needed to transmit the data.
We aimed to create a cheap and reliable method without the need to buy complex equipment and maintain expensive laboratories. The core value of the method is in its potential application ‘in the wild’, in situations where people move, talk, experience different emotional states.
We took six different trackers: Honor band 4, Amazfit Bip, Xiaomi mi band 3*, Apple Watch Series 2, Garmin, and Samsung Gear S3, to test how each tracker performed compared to (1) the real heart rate, (2) to Neurodata Lab’s algorithm for remote heart rate detection. We only used trackers with optical PPG technology. In total we made 75 video tests (1–3 minutes long): 8–10 in a calm state and 3–4 after a person went through some physical activity, doing squats, jumping, or pushing up. We considered that each tracker carried our measurements with a delay which is usually about 5–10 seconds, and adapted the results accordingly.
With or without previous physical activity, participants had to be sitting and wearing a bunch of gadgets, all for the sake of science: a wristband tracker and professional ECG equipment on the same arm — to compare the results to the actual heart rate. At the same time, they had to turn their face to a webcam, while their heart rate dynamic changes were measured.
After the tests were conducted, we estimated Mean Absolute Error (MAE) that represented the deviation from the real heart rate for each tracker and compared the values to our solution’s MAE. This is a parameter that allows measuring how many heartbeats the tracker is wrong compared to the reference value. We used professional wearable photoplethysmographic equipment to set the referent heart rate.
In the table below there are the results for each tracker compared to our remote detection algorithm. The smaller number corresponds to a smaller deviation from the referent heart rate, and thus represents better accuracy.
All in all, the trackers were on average 3.63 beats per minute wrong. Our algorithm was only 2.38 beats per minute wrong. (We applied Mann–Whitney U-test to show the improvement in our algorithm compared to trackers is statistically significant; with p-value = 0.000041 < 0.01 it really is.)
In the graphs below you can see how each tracker performed within a time interval compared to the professional contact equipment, which was also PPG-based, and our remote heart rate detection algorithm, rPPG.
We believe it really is a decent result for a technology that can detect heart rate with a simple webcam, just like the one in a laptop. It shows great potential for the application of remote heart rate detection. Eventually, we’ll be able to understand what happens inside a person even when there are not enough visible data and that’s impressive.
There are numerous applications for the technology indeed, apart from an in-built camera in a treadmill in a fitness center that constantly measures the exerciser’s pulse. It can provide a comfortable noncontact way to monitor infants and elder people in hospitals or at home. It opens great opportunities for emotion recognition technologies and stress detection, further enhancing their accuracy and broadening potential uses. It would even help better monitor NASA’s astronauts or workers on sites back on the Earth. And thanks to the ability to work with compressed videos there will appear more ways to use the remote algorithms for better healthcare, safety, and human well-being.
We welcome you to test Neurodata Lab’s heart rate detection algorithm yourself at a brand new demo portal open for everybody at https://emotionsdemo.com/ (via Chrome desktop)!
Authors: Olga Perepelkina, Chief Research Officer at Neurodata Lab, Kristina Astakhova, Evangelist at Neurodata Lab.
