Heartbeat sensing with a touchscreen
Heart rate monitors are usually sold as individual devices, or when the functionality is included in another product they at least require dedicated sensors.
However, everyone with a smartphone carries with them a 100–200-sensor array, and its capabilities are currently dramatically underused. A regular touchscreen has a grid of sensors that measure capacitance. When a user touches the screen with a finger, the capacitance of the finger, arm and body is sensed by the touchscreen. Now, suppose that some physiological phenomenon would result in changes in the capacitance of the touching finger. If the sensors were sensitive enough, maybe they could pick up a physiological signal? Like the pulse?
Unfortunately, the raw sensor signals are usually used only in the first processing step — at the touchscreen controller — which is a chip that reads the sensor values and outputs abstracted touch information (like number of touches, touch coordinates, and touch gestures). This information is then used by the phone’s operating system to interact with the display contents. I have been looking around for some time for touchscreen controllers that would allow reading raw sensor values in order to try out if the touchscreen could be used in collecting physiological signals, but hadn’t bumped into any.
So when a guy called optisimon found out that the Raspberry Pi touchscreen could be hacked in such a way that the raw capacitance values could be read, it definitely caught my eye: http://optisimon.com/raspberrypi/touch/ft5406/2016/07/13/raspberry-pi-7-inch-touchscreen-hacking/
Bits and pieces
All the code used in this project can be found in github:
https://github.com/hyvapetteri/touchscreen-cardiography
Instead of using Raspberry Pi, I was happy to use any touchscreen with the same controller and have Arduino UNO handle the I2C communication. The touchscreen controller in the Raspberry Pi screen is FT5406, which is part of a line of touchscreen controllers. I went for a smaller screen (because it’s cheaper) and got a 4.3" display which had a touchscreen controlled by a FT5206, a baby brother of the FT5406: http://www.ebay.com/itm/4-3-inch-TFT-LCD-Module-Display-I2C-Serial-w-Capacitive-Touch-Panel-Tutorial-/291854295216
I also needed a FPC connector adapter from adafruit to hook it up with Arduino: https://www.adafruit.com/product/1492
The screen was powered by Arduino’s 3.3V pin and communication was handled by the SDA/SCL pins which were connected to 3.3V by 4.7kOhm pull-up resistors.
While optisimon used the touchscreen to do capacitive imaging, i.e. use spatial information from the screen, I was looking to get temporal, timevarying data. Thus, the temporal resolution of the touchscreen is important — in order to measure heartbeats, I would hope to get a sampling rate of around 20 Hz (Hz = 1/s = samples per second). In the datasheet for the FT5x06 controller line, it is said that a full scan of the screen takes about 10 ms, after which the values still have to be read via I2C. Using only a part (20 sensors in the center) of the touchscreen, I was able to get up to a sampling rate of ~50 Hz, while reading all sensors slowed things down to ~10 Hz.
The touchscreen data was read by Arduino and forwarded via USB (with Arduino’s Serial) to a computer in csv format, where the data was analysed offline with python.
Show me the data
There’s also an IPython notebook in github.
A 35-second segment of raw data, with a touch event from 12–30 s, looks like this:
The touch event is pret-ty clear, but at least here there’s no heartbeat visible in the signal. Taking a closer look at a 10-s segment from 15…25 s and taking an average over all sensors at each timepoint, we get:
Looks promising, but is it just random fluctuations? Autocorrelation is a good way to check if there is actual periodicity in the signal:
It’s definitely a non-random periodic signal. The first peak is at 35 samples. With a sampling interval of 21.35 ms this translates to a heart rate of 80.3 bpm. I don’t have a proper heart rate monitor available to check the result, but going old-school by counting the number of beats in 15 seconds and extrapolating to full minute for bpm gave an estimated heart rate of about 80–84 bpm, so I’m quite confident in the result.
Compare these results to a 10-s segment in the beginning with no touch present:
Although the average signal waveform looks quite similar to the touch signal, the autocorrelation plot tells the story: no autocorrelation, so it’s just random noise.
Thoughts
Measuring the heartbeat isn’t anything new, but doing it with a touchscreen is. Dedicated capacitive sensors have been used to obtain a plethysmograph (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3304141/). Plethysmography refers to the measurement of changes in the volume of an object, but I’m not sure whether the heart beat signal above comes from volumetric changes or just the small movement of the finger (https://en.wikipedia.org/wiki/Ballistocardiography).
In addition to the purely nerdish joy of just being able to measure heart beat with a touchscreen, there are some actual advantages that could be achieved by doing this.
Obviously, someone (can be manufacturer or consumer) can save some money by not needing an extra device or sensor for heart rate monitoring.
In addition, it might make sense to combine the touchscreen measurement from the finger with the signal from a proper ECG sensor to get more detailed cardiac information, such as the pulse transit time (PTT) which could be useful for assessing blood pressure:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4512231/
https://www.ncbi.nlm.nih.gov/pubmed/21556814
All in all, I think it’s a demonstration of just one of the cool things that could be done if the raw sensor signals were available to the phone OS and smartphone applications.
