(PART 2)‘Fake News?” It’s ‘Fake Science’ that terrifies me
In Part 1, we looked at the vulnerability of respected, peer-reviewed scientific literature to deliberate fraud. But what happens when the authors’ integrity is not in question, but rather the integrity of the data so diligently collected?
Engineers have a term related to this called ‘GIGO: Garbage in, Garbage out’
GIGO means that however good your model or algorithm or data analysis tool or even A.I. might be, those tools can only produce results as good as the data going in, and if you have garbage data going in, you get garbage results and the resulting medical decisions, policies and laws you develop are flawed, just as the data was flawed. Often a lot worse.
There are lots of ways well-intentioned researchers can end up with false data. For example, one issue is when ‘Professional’ clinical trials patients routinely falsify information. Remember that FDA decisions about legalizing a medication, or AMA guidelines recommending a patient take it instead of a different treatment, all come from this kind of clinical trial data or similar. Not that easy to guarantee real, trusted info.
There are also lots of physicians throughout the US who (despite severe potential penalties) routinely falsify patients’ electronic medical records (EMR), often in ways that do no harm and may even facilitate better care, but that do result in the accumulation of incorrect healthcare data. Here’s an example:
Two hospitals (both alike in dignity, except for the standard open-back, lower-buttock length patient gowns) that both have electronic medical records systems. On Hospital A’s system, doctors navigating patients’ electronic charts (EMR) come to a page of 14 screening questions for, say, lung cancer that they are supposed to ask every single patient. They have to check a box next to each question to confirm they’ve asked it in order to be allowed to proceed to the NEXT screen where they can write orders etc. Hospital B does not have this page of questions in their EMR.
A doctor at Hospital A, using their electronic medical records system, has a patient for a 15 minute consult slot. Working as fast as they can, they ask the appropriate questions and do the appropriate exams and navigate through page after page of the online chart. ALL Hospital A doctors check EVERY box on that page every time — they HAVE to, to proceed to the next screen. But how many doctors do you think actually ask EVERY one of those questions EVERY time? Very few. With limited time and excessive reporting burdens, most Hospital A doctors only actually ask the screening questions for patients they feel are relevant to the questions; all the other questions they just check the box and move on. Not that big a deal, right? So what, you might ask? It doesn’t hurt the patient, and in fact it might help them get BETTER care since their limited time with the doctor is not wasted on irrelevant but required questions.
Well, that data gets duly recorded and reported and later, when people are trying to figure out screening programs work, they might note that all these patients who were screened (or were supposed to be) at Hospital A developed lung cancer at the same rate as other patients at Hospital B who weren’t screened because that hospital doesn’t use that screening tool in their EMR.
You might therefore conclude that the screening had no value and ‘didn’t work’ even though in reality BOTH groups of patients at both hospitals — Hospital B and Hospital A — didn’t really get screened. You’d be basing your decision on essentially imaginary data. Suddenly your ‘evidence based’ policy decision (or your A.I. and algorithms to diagnose or select treatments for patients based on Big Data) is garbage, since it was based on garbage data.
Yet much of this kind of data is simply accepted at face value because there it is in the EMR, the doctor checked each box next to each question so they MUST have all been asked…right?
Other reasons — and they are legion — for incorrect medical data reporting include medical records altered to cover medical negligence, discrediting a colleague or promoting one’s own career, for fraudulent insurance billing.
So….when global health care policies are being concocted, and doctors tell you they practice ‘evidence-based’ medicine, you definitely should reassured (because after all, you DO want decisions to be based on the best available evidence) but it’s also a bit sobering to remember just how vulnerable that ‘evidence’ actually is. Evidence means ‘that which is seen’ but sometimes what we see is only the tip of the iceberg…and we’re on the Titanic.
So keep a sharp lookout ahead, remember that the part of the iceberg that rips your hull open without warning is the part UNDER the surface, question the basis for policy decisions, and do your best to think critically about your data or the data your hospital or government is using, and verify your assumptions by talking to trusted sources of context when you can.
I have my own reason, aside from any data, for believing in vaccines. My strongest reasons for believing in vaccines comes from the Ngäbe-Buglé indigenous people of Panama whom our medical team serves. Vaccine penetration is relatively recent in this population (I have a couple of patients my age who were crippled by polio) but just about every patient I’ve encountered there in these rural jungle communities is very pro-vaccine. Exploring this, I’ve mentioned to Ngäbe friends that there is a growing anti-vaccine movement in the US and elsewhere, and what did they think about that?
They puzzle over it and then say ‘Is it because their families are too big there?” The only reason they could think of NOT vaccinating your kids was because you wanted some of them to die. That is their experience, living pre- and post-vaccines, of why vaccines are important. The generations at my age and above have lost a lot of kids, sometimes half. They have a very recent memory of life without vaccines, and THAT experience is evidence I can believe.
I’ll take that over suspect data any day.
