She Blinded Me With Science — or, 4 Tips for Critical Reading for the Internet Age
“Everything you read on the Internet is true.”
- Abraham Lincoln
On this blog we’re going to investigate a lot of claims that may — or may not — prove to be effective cancer hacks. We’re now the leaving the protected harbor of basic science (see Cancer Biology for Patients — Episode 1, Episode 2, and Episode 3 — and “What’s a Gene?”, “Welcome, Initiate, to the Central Dogma!” and “What’s a Pathway?”) and entering the murkier waters of bleeding-edge research and Internet rumor. It’s vitally important that we practice some basic seamanship skills before sailing, so we can avoid the krakens and sirens lurking there.
So how do we navigate the vast sea of often unsupported claims out there aimed at cancer patients? We all tend to look less critically at claims we’re inclined to believe in — we have to fight this tendency, because people pushing agendas know it and use it. We want to believe many of them because we’re always looking for an edge (a “hack” in this blog’s parlance) — but our time is limited and precious (especially if we’re in a high-risk category) so it’s very important that we not waste it chasing shadows down blind alleys. So let’s take a few minutes to arm ourselves with some skills to help us assess what’s believable and supported and what’s not.
So let’s take this email hoax as a really glaring example of Internet folklore that is just a complete fabrication and see how each of the tips below would help us avoid giving it any credence.
1) Vet the source — is the source even real and accurate? We’ve all probably seen the email circulating around that claims to be a “cancer update” from “John Hopkins.” First of all, the institution is called Johns Hopkins, not John Hopkins. Red flag right there. A quick search on Snopes.com for “John Hopkins Cancer Update” turns up an immediate hit. Snopes can often be a great first line of defense for really over-the-top-sounding claims, especially those that Aunt Louise forwards to you daily via email. But in general, ask yourself if you know where this claim really originated — if there is no source given, or you follow up on the source and it’s a dead end, be very wary.
2) Look at the credentials of the author(s) — If a guy who sells auto parts claims to have found “the” cure for cancer, I’m not saying it’s not a valid claim — it just had better be validated and replicated by someone with training in the scientific method. Not that impressive-sounding credentials are not always a guarantee either. MDs and PhDs are just people too — they are not immune from bias, axes to grind, professional jealousy and politics, etc. These kinds of cases are harder to check out, as we’ll see in my next post when I drill into an example. In the case of this hoax, usually no authors are actually cited, though it’s trying to steal the prestige of The Johns Hopkins University (my alma mater!), but never tells you who at “John Hopkins” is making the claims. Red flag — there is no reference for you to check.
3) Extraordinary claims require extraordinary evidence — the more dramatic the claim is, the more evidence you should demand — if none is offered, or it’s very tenuous, it’s probably a hoax or overblown. The “John Hopkins” email simply provides no evidence at all — presumably in the hopes that you’ll fall for the prestige of “John Hopkins” and not be too demanding.
4) Check the citations — first of all, are there any? One can claim whatever one wants (see my “Lincoln” quote above!) but without citations, there is no way for us, the readers, to know where the claim originated, so we can check up on it ourselves and see if we agree. Does the author footnote important claims with references to scientific papers, etc.? Are these citations from reputable, peer-reviewed journals? (“Peer review” is a formal process where other noted scientists from the same field first check out a paper with a critical eye and make sure the rules of science were followed before allowing the paper to reach publication.) Do the citations actually relate to the claim at all (or worse yet, do they actually contradict it — I wish I had a dime for every time I’ve seen this — it’s either profound carelessness, misunderstanding of the source, or willful misdirection on the part of the author — none of which are good). Sometimes blog authors seem to throw around citations after just having read the titles, without attempting to digest the contents. You don’t have to be a scientist to do some basic evaluation of scientific papers — see “How to Read Scientific Papers and Journal Articles.”
Also, there are huge pressures to publish in the academic world — both in terms of volume and “getting there first” — so the literature is full of papers with rushed and and/or contain weak or questionable conclusions. This isn’t always bad — it is sometimes part of the process of getting exciting scientific ideas into circulation for evaluation, and as long as the paper clearly states that the authors know the conclusions are weak, trained scientists know to take these with a big grain of salt (and possibly go try the ideas out for themselves). Often these papers conclude with something like “While the findings did not rise to statistical significance, this is an interesting line of inquiry that bears further study in the future.” The sad thing is, then some blog author gets ahold of these citations and uses them to “validate” a point that they actually don’t support. Unless you, the reader, then follow up and read the citations, you’ll never know this — and you think, “wow, that’s impressive — someone at Harvard agrees with this point” — when in fact, they may not at all.
So, if we identify a poorly-supported (or completely unsupported) claim — does that mean it’s false? Not necessarily. It just means that the author presenting the material to us didn’t do his or her homework (or worse, is deliberately ignoring or misinterpreting evidence). If we really want to believe this claim, we’re going to have to find a better argument for it. Now, if we find that the evidence is against the claim — then we should be very wary of expending more energy on it. There are too many other ways we could be using our time.
