In this collection I want to explore how things go wrong, and this example provides a great starter, since it involves lots of smart professionals, subject to lots of checks and balances, working very hard, in public, who nevertheless manage to produce a deeply and consistently flawed product that is likely hurting the welfare of millions of people:
The popular health science press.
David H. Freeman, writing a tour de force for the Columbia Journalism Review this week, shows how prestigious publications routinely blow it in their coverage of medical fads supposedly supported by peer-reviewed science. Diet research bears the brunt of his criticism, and he cites numerous examples of contradictory theories and conclusions within the pages of one and the same publication, backed up by a mishmash of contradictory tests seemingly cherry picked for the arguments at hand.
It’s easy enough to verify that something is going wrong with medical studies by simply looking up published findings on virtually any question in the field and noting how the findings contradict, sometimes sharply. To cite a few examples out of thousands, studies have found that hormone-replacement therapy is safe and effective, and also that it is dangerous and ineffective; that virtually every vitamin supplement lowers the risk of various diseases, and also that they do nothing for these diseases; that low-carb, high-fat diets are the most effective way to lose weight, and that high-carb, low-fat diets are the most effective way to lose weight; that surgery relieves back pain in most patients, and that back surgery is essentially a sham treatment; that cardiac patients fare better when someone secretly prays for them, and that secret prayer has no effect on cardiac patients. (Yes, these latter studies were undertaken by respected researchers and published in respected journals.)
How do celebrated expert reporters, vetted by professional editors, citing research papers published by leading science journals, wind up making contradictory, misleading and — in retrospect — often outrageous claims from the outer fringes of the research world?
The answer is deceptively simple: Lack of proper skepticism. According to Freeman, science reporters have come to believe in a myth of the infallibility of science and no longer pose tough questions to researchers, relying instead almost entirely on press releases — in contrast to their peers on the political and business beats, who don’t hesitate to fact check, verify and test the statements of officials and executives in great detail. (The comparison seems too flattering to the latter; as the Manti Te’o story takes over the news this week, he is almost certainly overestimating the innate doggedness of the general non-science press.)
If you liked this post, feel free to join this collection with your own examples that highlight, explain or otherwise peel back mechanisms of failure.