Wearable Biofeedback (Accelerometry)
Designing for Wearables, Advanced Design Prototyping, Accelerometer-Based Motion Estimation
Defining The Problem
I was tasked to design a new “on-demand health reading” experience for a health-tracking mobile app: where users could now get a health reading (let’s pretend it was IQ) using their wirelessly-connected wrist-wearable.
While I also worked on the end-to-end flow (user education, data visualization, results & summary, etc.), the work described here is solely focused on the experience that takes place during a reading, until it ends (or fails).
Challenges
This breakthrough in wearable health-tracking technology presented an interesting set of constraints and challenges:
- A reading could take between 1–3 minutes…
- Readings were extremely sensitive to a various of factors. And even slight movement could compromise or prolong the reading. Ideally, users shouldn’t walk, move their hand, talk, laugh, or even breathe irregularly during a reading.
- Unlike most health-tracking gadgets, our wrist-wearable lacked audial or physical signifiers that could indicate whether a reading was in progress.
- Our app must remain foregrounded for the entire reading duration (or else the data would be compromised).
To me, the biggest challenge was designing for a system that could finish readings in an variable amount of time. Combined with our lack of signifiers, I wondered: What’s to stop a user from raising their hand mid-reading (ruining the reading) to ask “IS THIS THING STILL ON?”
In hypothesis form, my goals for the design were to:
- Reduce the % of motion-related failed readings
- Reduce the average time-spent-waiting per successful reading
I believed that succeeding at these goals would best promote long-term engagement with this new feature.
The Design Process
Group brainstorming and early sketching gave me plenty of concepts to start with. Common themes started to emerge, and I grouped them along these categories, along with their pros and cons:
- Gamification: engaging users in some game that incentivizes user-stillness.
- Follow Along: guiding users towards a desired behavior through some activity — like a breathing exercise or a guided meditation.
- Look, A Distraction: distracting users by providing content (educational, humorous, etc.) to consume.
- “Super” Loaders / Progress Bars: visually-engaging animations that convey progress or system status.
While I’m sure users would welcome any improvement to a boring, simple countdown timer or a looping animation, it was compelling to wonder if the last approach — if executed well — could actually shift the user’s perception of the experience from being “slow, unresponsive, and dumb” to “complex, cool, and cutting edge” in a way that would compliment — not detract — from our story of “breakthrough innovation”.
Divergent Explorations & Prototypes
First, I explored the idea of a dynamic avatar that’s visually reactive to the system’s status.
While it’s dynamism was more informative and visually compelling than static text and, it wasn’t helping users avoid receiving the errors in the first place.
To address this, I explored how I could help users avoid moving too much before an error occurs, instead of after when it’s too late. Vague textual instructions (like “Please Stay Still”) — when used alone — struggle to capture the exact level of sensitivity of the system. One compelling idea was to perhaps provide a real-time feedback loop — one that could help users quickly calibrate to what the “acceptable amount of movement” was, and perhaps even warn them if they were about to cause an error.
This led me to explore how I could imbue real-time motion feedback into the experience.
Here, I used an ribbon as a visual metaphor for the reading’s ‘data stream’, where moving too much would visually ‘destabilize’ the signal.
Guerilla user testing of this design revealed that users were intentionally moving more rather than less, because they were either intrigued or entertained by how their movement was directly impacting the (rather unfamiliar) visualization. Despite this, we saw that users quickly adjusted to the extreme level of stillness required for the rest of the reading, and it was thus a promising tool for preventing motion-related errors.
To avoid overly-distracting users with the motion feedback, as a team we decided to simplify the visualization and emphasize the progress of a reading as a primary design goal, via a radial progress bar — and then find ways to incorporate real-time motion feedback more subtly:
These considerations were important to me as I continued to iterate:
- The intensity of the visual feedback should be proportional to how much the user is actually moving, rather than some binary error state
- The feedback should help users quickly learn what is or isn’t an acceptable level of movement. Not all users would expect that flipping over their hand (to use their phone for example) would be considered too much movement.
Fine Tuning The Experience Towards Production
As we were starting to converge in on the final concept, I was prepared to wear multiple hats and get my hands dirty with the implementation details.
New Tools For New Problems
First, I needed to calibrate the fidelity of my prototypes to the current stage of design I was in: vigorously shaking my phone to demonstrate the effects on my laptop was appropriate for communicating a design concept, but was far from resembling the actual user experience.
To bridge those gaps, I sought out the necessary collaborators across our Firmware and Software engineering teams to rethink & improve our prototyping process.
The result was a special iOS internal app that extended on my existing setup: this app (1) connects with an actual wearable device, then (2) sends accelerometer data from the wearable, to the app, then to a Firebase cloud database, which (3) multiple Framer prototypes could connect to in real time. This accelerated our design process in a couple of ways:
- We could test our designs in more realistic scenarios (a user wearing a band while holding a phone running a Framer prototype)
- The speed at which we can design and prototype using wearables and any “new band signals” in the future, would be less dependent/bottlenecked by limited engineering resources
Reverse-Engineering A North Star: Algorithm Development
Extracting meaning from a “firehose” of raw accelerometer data was far into uncharted territory for the design team, but I was excited to dive in and wrestle with this new technical challenge. After all, you have to truly understand a thing to design a thing.
“You have to truly understand a thing to design a thing. Imagine trying to design a plane, but not knowing anything about aerodynamics, or designing a glove without knowing what environment it will be used in or the anatomy of a hand. You have to understand what something can do in order to design a product well.” — Amber Cartwright (Airbnb)
When Googling couldn’t get me directly there, I dived into everything from scientific literature, online tech-talk videos, to product forums (Myo, Processing, Android) in search for answers — fumbling through foreign concepts like digital signal processing, sensor fusion, dynamic time warping, and integration drift along the way.
Through this, I eventually made a simple motion estimation algorithm that sufficiently detected subtle band movements (not orientation or position) using only a tri-axial accelerometer. These were some key developments that occurred in the process:
- I initially tried comparing the differences in acceleration magnitude (sum of squared vectors), but it failed to detect the motion from rotating in place (i.e., turning a wrist over — where individual vectors change but the overall magnitude stays roughly the same). This led me to compare the collective differences of individual vectors. (I eventually learned that this method was called Sum-Absolute-Differences, which apparently is frequently used in fields like digital image processing!)
- Initially, I just compared the differences between the two most recent accelerometer values. But so little time passes between one accelerometer value to the next, that comparing those differences wouldn’t be very meaningful. This led me to implement a combination of (1) a moving window to smooth out irregular fluctuations and understand longer-term trends, and (2) a simple moving median (preferred over a moving average because it’s less sensitive to outliers) to provide a better baseline to compare the most recent accelerometer reading against.
- Additionally, I calculated orientation (pitch & roll) for tilt sensing from accelerometer data to use as an input.
The Result
After more iterating, I collaborated with a visual designer for further refinement on visual and choreographical elements. In the end, our design combined the use of visual, animatic, and textual elements to guide users throughout the reading:
- A “content card” would sporadically surface textual educational content to help entertain the user.
- A radial progress bar visualizes how much time is left in the reading.
- There are three “motion thresholds”: (1) for very slight movements, the circle of dots starts to jiggle; (2) for movements that are moderate enough to negatively affect the readings, a textual warning is displayed, with a proportionally darkened background, and more rigorous blob motion; (3) for extreme levels of motion, the reading quits.
And just for some office fun…
