Blokeology | Four lessons in high intensity skepticism
You might think that because a paper makes it into a reputable journal it has certain irrefutable qualities. Certainly, I can reflect on my own involvement with medical journals, most recently at the BJGP, and plenty of efforts are taken to ensure a paper is high quality. I’m not foolish enough to think that the process is flawless and, undoubtedly, there will be aspects of a paper that aren’t perfect.
Indeed, I would argue that almost all papers are flawed in some way and there is no such thing as ‘perfect’ research. Those are largely minor and inherent to the particular research technique used but they do exist. My own policy as an editor is to make absolutely sure that those limitations are acknowledged and discussed within the paper. That’s all part of the process.
There’s an interesting example with a recent paper, that clearly has some challenges, in the British Journal of Sports Medicine that made me think a little more about my own skepticism.
Skepticism lesson #1: Check your biases.
Back in Episode 036 of the Blokeology podcast ° the excellent Dr Michelle Swainson came on and talked about the evidence for high intensity interval training, hereafter called HIIT. For many years I’ve had a rather negative view of HIIT and I had always thought of it as a rather faddy exercise programme, designed to appeal to the countless numbers of people who want quick and easy fixes to their problems. Michelle rather set me straight. It turns out that in all the years since it first appeared there is a growing body of evidence, including, and this impressed me the most, studies in people with co-existent disease. It was an object lesson, and one that is particularly important as the years drift past. It has been over 20 years since I left medical school and that’s plenty of time for evidence to emerge. You may think you are the model of rational thinking but you are not. We all have very human biases.
So, I approached a BJSM paper on HIIT with an open mind. even though it was headlining some extravagant claims.
Skepticism lesson #2: If a headline is posed as a question the answer is usually no.
A recent important study in the BJSM° showed some impressive fat burning properties of HIIT compared with standard continuous exercise. It was given the natty pre-title: Is interval training the magic bullet for fat loss? The academic bit of the title went on — A systematic review and meta-analysis comparing moderate-intensity continuous training with high-intensity training (HIIT). Now, for a kick off, it’s a good skeptical rule of thumb that any time you read a headline in a newspaper (or medical journal) that is phrased as a question the answer is usually ‘No’.
The findings of this paper rather suggested the answer could be yes. The good news is that both HIIT and moderate intensity exercise will help people reduce total body fat. (That’s assuming that’s your aim of course. We won’t get into arguments here about the healthy overweight — though I think, overall, we could agree that many people would benefit from dropping some body fat.) The headline figure is 28.5% which is the increase in reduction in body fat found in the HIIT groups. Sounds impressive and indeed this was statistically significant. Here’s the forest plot.
That doesn’t look quite so startling. You can see just how closely that bottom diamond is nudging the confidence interval. There’s a lot hanging on that diamond not touching the line. It wouldn’t take much for it to cross into non-significance. And, arguably, this is not a binary process either — we all need to be a little careful here about drawing radical conclusions for clinical practice.
Skepticism lesson #3: Published does not mean perfect.
Now, the forest plot thickens.
The paper was published in May 2019 but a month later the BJSM took the unusual step of publishing an ‘Expression of Concern’° triggered by an eLetter ° about the data extraction and analysis. Intriguing. You can also see, at an easy glance, that the Thomas et al 1984 paper (the top one) has a monster impact. The Thomas et al paper apparently showed a remarkable difference of -13.4kg in favour of HIIT between the two groups in a 12-week period. Which is an astonishing difference. Moreover, the authors of the eLetter suggest this 13.4kg difference was never published and didn’t come from the original author. Was it calculated afresh from body fat percentage data? We don’t know. In addition, the authors of the eLetter also question the inclusion of Boer and Moss, 2016 which is also slightly tilted towards HIIT.
If either of these papers were to be excluded then, in short, the wider suggestion about HIIT being the ‘magic bullet’ looks highly contentious.
Which is a good reminder that just because ‘evidence’ gets into a scientific journal doesn’t mean it is set in stone — it can flawed for any number of reasons. And, it seems, some further information about the analysis came to light after publication. Post-publication processes are important. Now, it is still possible that there is no problem with this study but the BJSM has asked the authors to address these concerns and provide some amendments. At the time of writing that was still awaited.
Skepticism #4: The damage is already done — and continues
The problem is that this paper got a lot of attention. I’d argue the authors courted this with their “magic bullet” phrasing in the title. You can get an idea of how widely is was disseminated from the Altmetrics for this paper° which goes some way to quantifying the attention it has received. You can look at the details of these news articles, blogposts and tweets and they are almost all promoting the idea of HIIT to help weight loss. Worse, many of them have been published after the expression of concern was published.
We don’t actually know if the results of this analysis will change the final conclusion. It should be said that it looks highly possible — that is, after all, why the editor has released an expression of concern while stopping short of retraction. That’s all in line with the Committee of Publication Ethics retraction guidelines.°
Meantime, it is still being shared despite the presence of that note. The problem is that few people will be aware of the significance of an expression of concern and that’s assuming they read it at all.
In the world of sharing and social media the processes feel rather inadequate.
Photo by Massimo Sartirana on Unsplash
Originally published at https://www.blokeology.io on August 11, 2019.
