Science Is Never Perfect
Why bias in studies doesn’t make them worthless
There’s a pretty fundamental misunderstanding that most of us have about science. I remember being a slightly-younger Health Nerd sitting in a university tutorial and trying to work my way through a study on mesothelioma (nasty cancer) in asbestos miners (people who get nasty cancer).
If you’ve heard the scary word asbestos, you can probably guess what the study found about the people who mined it. They got lots and lots of nasty cancer.
And you know what the most interesting part about this study was?
It wasn’t done perfectly. In fact, it was done in a way that was pretty wrong.
There’s a misconception that we are all often prey to when we look at scientific evidence. It’s something both painfully and wonderfully human; we look for a black-and-white answer. We are simply incredibly uncomfortable when we look at something and it gives us an answer that is in shades of grey.
And the thing that no one really likes about science, the thing that makes almost every article you’ve read about science wrong, is that science is almost never about perfection, particularly in the world of health. Outside of extremely theoretical research, most scientific studies are far from perfect.
But this doesn’t make them wrong. It just makes them more difficult to understand.
Ibuprofen And Heart Attacks
I recently wrote an article about ibuprofen and heart attacks. It was in response to a media furore that had thousands of articles from across the world whipping people into a frenzy about the ibuprofen that was basically just slaughtering them in their thousands.
In my article I pointed out the deficiencies in this study that made it largely useless for individual decision-making; odds ratios, correlation vs causation, all the finicky little things that are hard to understand until a nerd explains them to you in detail.
This study wasn’t perfect.
But you know what? It was pretty damn good. It just didn’t say what the media thought it said.
I’ve now seen people saying that we should throw the results away. That’s incredibly stupid. Not just because it was an excellent study that had some important findings, but because there are virtually no perfect studies.
Science Isn’t Perfect
I can count on the fingers of one hand the number of perfect studies that I have ever read. Studies where the answer was absolutely defined by the data, because the study was so perfect that no possible errors could’ve crept in.
And when I say “on the fingers of one hand”, I mean one. One study that was 100% perfect in design, method and execution. If you’re interested, it’s this study that found paracetamol to be ineffective at treating acute lower back pain. It’s an amazing case study in brilliant research.
I work in public health. I’m usually running between five and eight studies at any one time. And you know what?
They are messy, problematic, and give me endless headaches.
Funding is an issue. Practical considerations are an issue. How do you test a new treatment for self-management of diabetes if it requires a smartphone in a population of old people who don’t always have smartphones and you can’t afford to buy them for the study? What about an app that was developed by a Taiwanese company, has been (poorly) translated into English, in a multicultural population that doesn’t speak either English OR Mandarin?
Science doesn’t have to be perfect to be right.
Suffice to say, it’s not easy.
But all of the research I do provides useful evidence. There are endless caveats, and you have to take it with a grain of salt, but I can still give you some really interesting insights into diabetes self-management.
Science doesn’t have to be perfect to be right.
And that is something that we are not very good at accepting.
It’s always fun to write a shouty article about the latest scare story in the media, and I do love laying out exactly why the research has been misinterpreted and doesn’t mean what you think it does.
But that is very different from arguing that it is useless. Most of the studies that I write about are actually great science and really well done. They just aren’t perfect. They have problems, potential sources of bias, issues with design, and a thousand other minor worries that mean you can really pick them apart.
But when push comes to shove, they are almost all brilliantly useful. Even when they find nothing, because nothing can be important.
In science there are rarely easy answers. Most studies give you a wishy-washy answer that is full of caveats.
There’s usually an entire section of the study where the authors tell you everything that could’ve possibly gone wrong with their study and how that might lead to a different answer than the one that they’ve given.
But that’s why we all do these pesky degrees and spend years learning analytical skills. Anyone can read a study, but putting it into context and understanding how and why it is meaningful can take years of effort.
I’m not saying you shouldn’t be analytical, or that you should give up on picking apart studies yourself. But next time someone offers you an easy answer, a black-and-white solution to a range of greys, remember:
Science is never simple.
There are no easy answers.
If it doesn’t give you a headache, you’re doing it wrong.
If you enjoyed, take a second to click on the heart below to let me know! You can also follow me here on medium, on twitter, or on facebook. If you didn’t enjoy, I’d love to know why in the comments, or @ me on twitter!