Texas Sharpshooter fallacy made large
Academia has a fascinating issue I’ve been pondering about recently. In essence, the fact that almost all journals have a clear and obvious bias towards positive results, there is almost a guaranteed texas sharpshooter fallacy that will occur over time when many many papers are written.
From wikipedia:
The Texas sharpshooter fallacy often arises when a person has a large amount of data at their disposal, but only focuses on a small subset of that data. Random chance may give all the elements in that subset some kind of common property (or pair of common properties, when arguing for correlation). If the person fails to account for the likelihood of finding some subset in the large data with some common property strictly by chance alone, that person is likely committing a Texas Sharpshooter fallacy.
Now, I’m not entirely sure if this is the best fallacy to use, but basically it’s based around the idea that given a million monkeys and an infinite amount of time, someone will eventually write Shakespeare. Now, if we ignore all of the negative results, and all of the gibberish and only publish once someone writes Shakespeare, it would seem, at times, as if a monkey left to a certain type of typewriter is more inclined to write Shakespeare.
Don’t get me wrong though, science is explicitly built around the idea that these points would be made public and then eventually proven false by repeated experiments by other individuals. However, with the modern media hyping everything the moment it is published and an overconfidence on anything that is “science” or “published” we have a big problem on our hands.
Yes, evolution is a fact, that’s been around long enough and there is no credible theory that challenges it that I’m willing to say that. In the same sense that I am wiling to say that Maxwell’s Laws are facts and the like.
However, 90% of the new science you keep hearing about… Is probably wrong. Especially psychology and medical science has this flaw (with the sheer number of papers and the need to publish or perish, how could it not.) Yet, we keep going back to that trough. It was published in a peer-reviewed journal, so it must be true, right? The media hypes that red wine is good for you, bad for you, indifferent…
Here’s my theory on this whole matter, not peer-reviewed or published, but at least a functional heuristic. Give it 20 years, or 40 years. Let people challenge the results and see if they actually apply. I wish we could teach people how science (and to be honest any knowledge development) really works. You have an idea, it seems to work for you. You try to figure out why and reproduce it so you can continue to enjoy the benefits. You develop something that seems to work and is reproducable, then you tell people and they try it out.
And 9 times out of 10 they discover that you weren’t quite right.
However, if the idea continues to generate positive results for you, you ignore them and keep using it. If they are right and it starts to fail, you go back and you try again.
To believe that new science is right simply because it’s published is to be as dogmatic as to believe the a religious book is correct simply because someone told you so. Journals (even good journals) are not always right.
