It’s not time to throw away your Fitbit, but it is time to rethink how to do research.

Yesterday the internet spread some exciting new findings in JAMA like wildfire — the story goes that wearables might actually harm your ability to lose weight.

We had big name press organizations all jump on board:

* Time: Your Wearable Won’t Help You Lose Weight
* PBS: The average person is better off without a fitness wearable, weight loss study finds
* NYT: Activity Trackers May Undermine Weight Loss Efforts
* Guardian: Fitness trackers may not aid weight loss, study finds
* NPR: Weight Loss On Your Wrist? Fitness Trackers May Not Help

…. just to a name a few. FWIW, MobiHealthNews got it right.

The problem? The device used in the intervention bears no resemblance to a “wearable” as we’d call it today. It was this device, a discontinued, very clunky, very clinical looking arm strap and sensor brick called the Bodymedia FIT Core:

Now discontinued device, represents the “wearable” in the Jakicic et. al. study.

Even the amount of data collected in the study suggests that the term “wearable” may be a bit of a stretch, as on the days that participants wore the device, the median wear time was 4 hours (241.1 min/d to be exact).

To equate the user experience of today’s modern wearables, as many have done with this, got me a little sarcastic…

I was certainly frustrated. This isn’t the first time this has occurred either. Why does this keep happening? And, more importantly, what does this say about the media, journals, researchers, funders, and the clickbait articles we’ve come to enjoy consuming.

The Media

This is an easy target. But, I do think we have a right to expect better of these reporters. What happened to scientist journalists who would have taken a skeptical eye and actually read through the paper? Did any of those writers take the time to dig through the supplemental material, which included information on usability and device wear behavior? Who signed off on putting a Fitbit as the leading image on these articles? Who wrote the title? Certainly not someone who read the paper. I find the Weight Loss On Your Wrist? Fitness Trackers May Not Help the most egregious title because no one in the study actually wore a device on the wrist. We should expect more from the people tasked with helping inform the public about advances in research and science. Ironically, I believe the commenters over at the New York Times website may have done a better job addressing the nuances of wearables, health, and weight loss than the author:

“I think that one cannot draw the conclusion from this experiment that it is counterproductive to wear a fitness tracker- after all, there is no control group which was simply urged to exercise but was not required to log in on a daily basis their activity.” — clark

“Was it that the fitness trackers were inferior, or that the manual exercise log was superior? Without a group C that didn’t have to log their exercise and diet, how can we know?” — R.V.S

“The armband used in this study is not the most user-friendly device. Not by a mile. That in itself is a very plausible explanation for the results. It’s frankly irresponsible for the Times to use a photo of a Fitbit with a sweeping headline (‘Activity Trackers May Undermine Weight Loss Efforts’) when the device used was NOT a Fitbit, nor even a wristband, nor was it anywhere near as convenient to use as most of the devices that people think of when they hear ‘wearables.’” — LB

The Researchers

Jakicic et. al. ambiguously refer to the “wearable device” repeatedly in their paper while only making one small reference to the model. Additionally, they address the overwhelming limitation, and difference it has with wearables that are used today with a single sentence:

The multisensor wearable device was worn on the upper arm, which may not reflect the effectiveness of more contemporary devices worn on the wrist.

Otherwise, the study concludes:

Among young adults with a BMI between 25 and less than 40, the addition of a wearable technology device to a standard behavioral intervention resulted in less weight loss over 24 months. Devices that monitor and provide feedback on physical activity may not offer an advantage over standard behavioral weight loss approaches.

At best, this conclusion is overstated and at worst, it’s a little sneaky. By using the ambiguous term ‘wearable device’, the author is, intentionally or not, allowing the reader to use their modern definition for the word, but are actually referring to a device that we wouldn’t really hold in the same class as what we’d consider a modern day ‘wearable’. That’s exactly what happened, the media headlines illustrate that. When we read about social networks we think of Twitter, Snapchat and Facebook, not MySpace and Friendster. Furthermore, at no point do the authors inform the reader that the device they used in the study is no longer available, having been discontinued years ago, or address the actual differences in currently available wearable devices (and their applications) and the device in the study. And before you say there isn’t much research on currently available wearable devices, spend some time over at PubMed and see what you can find.

The overarching generalization regarding the utility of “wearable devices” is probably the most egregious error on the part of the authors and the media. By studying one tracking device, and then concluding that “wearables” may not work, the authors make a leap that is at best poorly worded and at worst a gross attempt at framing their work to fit into the modern consumer landscape. And let’s be honest, there is a big difference between what was available to consumers via the BodyMedia experience in 2010 and what is available now through popular wearable devices and applications. For instance, did you know that in order to actually see the data about your activity from the BodyMedia FIT Core you had to wear another wrist device? Yup, two devices just to see how many steps you’ve taken.

The Funding Mechanism of Research

Taking it a step further, we’ve got a real problem with the pace at which we do research. This study is just being published now, in 2016, but it began in 2010. This represents a big problem. The time it takes to get funding for a study and then get results is simply much too long for the results to be relevant to the current technological landscape. Randomized controlled trials might be too slow for mHealth.

But if you’re the investigators of this study, what do you do with these findings? Do you say, “Well, the device looks nothing like anything on the market today, and our findings are probably tied pretty strongly to the user experience of a device no longer in existence, so let’s just publish these findings tied specifically to our outdated intervention and move on.” ??? No, you don’t say that. Because no journal will accept that paper, and it certainly won’t help with your next grant that your last big one didn’t make a big splash.

There’s at least two problems here with how researchers get funding:

* There’s likely too much emphasis on RCTs. Look what this RCT gave us.
* There’s a catch-22 at play in a system that rewards funding to only to things that succeed. Failure is part of learning, but success gets grants.

The Journals

Journals are an important part of the process, but they also have an agenda. And it really depends on the set of reviewers and editors you get. In this case, it’s a shame a reviewer didn’t push them harder on specific language on how the wearable tested compares to what a reader in modern day times might interpret it. It’s also a shame a reviewer didn’t push harder on the 4 hour wear time figure, because most people today are looking to better understand their behavior as it relates to all-day activity and trends over time. These are not insignificant differences, and to allow the comparison feels like JAMA (a usually very credible and a very important part of the narrative of this field) let one slip past the goalie.

Us — How We Consume Knowledge

We’re increasingly less able to look at nuance. We want the big, quick hit of digestible tidbits. From science, to politics and policy, to religion and ethics, it seems we’re ever so increasingly looking for black and white. Quick clickbait gives us this. We need to read longer articles. We need to look at primary sources with a sense of skepticism and make them stand up to the claims they seek to make.

Wrapping It Up

There’s enough blame to go around, but in the end, it’s time to start turning this ship around. We need faster grants. We need more early career researchers getting pilot and even larger studies funded. We need studies getting early findings out in 6 months or less. We need to be OK with results that don’t come from RCTs. And we need innovative tools, methods, funding, and frameworks that encourage fast learning, support failure, and disseminate this in simple ways to the public. We need more researchers thinking about Agile Science methodologies. Longer running studies, especially utilizing relevant technologies, and pursuing innovative and adaptive interventions, which are needed to keep up with modern versions of the devices and systems they’re trying to test. We also need to push researchers to publish their findings as they go, in smaller increments, more quickly, and more readily accessible by the public (in both access and language).

I’m proud of being a toolmaker in this space. My company places an emphasis on supporting small, nimble pilots, even when we only break even on many of these projects, because we care about supporting innovative research. It’s an important part of moving this field forward, but we all need to take a little responsibility and make sure science is about chipping away at finding truth, not writing headlines.

Disclaimer: My company, Fitabase, makes data tools which make Fitbits awesome in research. And I am, at times, critical of the field I take part in because I care deeply about it. Thanks to Ernesto Ramirez and others for help drafting this post.