The numbers don’t lie
or “Why most statistics are complete crap” and why that should matter to you
I'm no statistician. I am a biomedical researcher though (or at least studying to be one) and one of the first things I learned from graduate school was that people twist their data to reach the conclusion they want all the time. “Trust no one” is probably the degree I should receive instead of Translational Biology once my time is complete, because it’s probably the most significant lesson I’ve learned to date.
Statistics are used in science to show that your data was “significant” — meaning that it’s unlikely the results you received were due to chance alone. Only problem is that you can alter your parameters until you achieve a significant result and this happens all the time. Scientists, like all people, have an inherent desire to succeed. When you’ve spent 6 months, or a year, or 3 years collecting the data and analyzing it only to find a non-significant result… well it would be tempting to tease the data to find the significance. I’m not talking about outright fraud here, but perhaps someone might reduce the effect size expected, or increase the N of each group, or choose a different statistical test. As my stats professor told us “Torture the data long enough and eventually it will tell you whatever you want”.
In fact, this problem is so pervasive that it’s made its way into the mainstream consciousness. The New York Times, Reuters, and the Economist (to name a few) have all published stories on the alarming lack of reproducibility and rise in retractions over the last few years, causing a serious rattling of the public trust in science. This is bad news people. If you are one of the people who shake their heads wondering, “Why do people still question evolution/climate change/insert hot button politico-scientific item here when there is an abundance of evidence?” well this is why. When the public is exposed to so many tales of scientists behaving badly, it is quite easy to make the cognitive leap from isolated incidents to institutional ignominy. And, if they were even the slightest bit well read on scientific issues, they would quickly realize that the problem is more than isolated incidents. So it’s no wonder that when the issue is polarizing, it is so easy for people to dismiss their opposing side as being biased or just plain wrong.
It’s not just science that has this problem though. Politics is notorious for this sort of malfeasance. Just think back to… oh… any political poll ever taken ever. Depending on which network, newspaper, or pundit is talking your candidate could be souring or soaring in the polls, your position the rational view of the majority or the lunacy of a vocal minority. Personally I only take Gallup, Pew, and Rasmussen polls seriously, and even then it is with a grain of salt.
One of the most egregious examples I have seen recently (and yes this story is a few months old, however as the women in question is now running for Governor, it has brought the story back up) was a lauded (by progressive groups anyhow) poll that was reported to show that 80% of Texans opposed SB5 (anti-abortion legislation).
But is that really what it showed? Not so much. In fact, this poll wasn’t a legitimate objective opinion poll at all (which the participants probably assumed) but something known as a “hired gun poll”, also known as an “advocacy poll” — and could even be said to stray into the realm of the “push poll” with some of their questions.
More than anything, what the response to question 5 says to me is that Texans believe education, jobs, and the economy to be more important at the time than the abortion issue. It does not, however, mean that the participants fundamentally disagreed with the legislation. In fact, when participants were directly asked about their opinion of the bill the results looked a little different:
There are several fundamental methodological problems for this survey from my perspective. First off notice that each question’s prompt mentions that the question is being read aloud by the pollster from GQRR. No big deal right? I’m sure Gallup and Rasmussen does the same on occasion. Except that Gallup and Rasmussen do not have demonstrated conflict of interest between their poll results and their clientele. It is obvious, from a 5 minute perusal of the GQRR website, specifically their client list, which direction the wind blows in those offices. Planned Parenthood, Emily’s List, Pro-Choice Voters, NARAL Pro-Choice America, Women’s Voices. Women Vote… and yet strangely there are no Pro-Life groups in their roster. Hmm.
So, it’s highly likely that the questioners could use the tone of their voice to influence the responses that they receive from the participant. In fact, this is a well-documented phenomenon in which the interviewer impacts the results of the survey they are observing and has been published on as early as 1944.
But let’s give them the benefit of the doubt and assume that the interviewer kept her tone completely neutral — problem solved, right? Wrong. Take a look at the highly loaded language just peppering this survey (including in Question 10 shown earlier about the specific proposal up for a vote and the following examples below)
In the examples above, participants were asked to select which of the two statements they felt best fit with their opinions. I haven’t mentioned it yet, but the survey also included information about the self-identified political tendencies of the 601 participants (all registered voters, so there is some sampling bias), and 46% identified as conservative, 33% as moderate, and 15% as liberal. Oh and 67% of them said they followed politics closely (cough… cough.. bull$h*t … but let’s go with it.)
Now, they didn’t ask directly, but if I had to guess (because it is Texas) a good proportion of those participants would at least somewhat agree with a statement somewhere along these lines:
The government has little impact on my daily life and I do not feel they need to create any new laws or regulations because we have enough already. The government exists to serve me and not the other way around.
Notice that in each of the opposing statements, the one chosen by the greater percentage of the respondents favors individual determination and less government intervention/regulation/law-making. So it is impossible to draw any firm conclusions regarding Texans’ opinions on abortion restrictions because of the confounding variable of “don’t tread on me” libertarian sentiment.
In fact, Gallup polls have shown that as much as 20% of voting adults identify as libertarian — a group that agrees with the economic sentiments of conservatives and the social policies of liberals. You could even argue that these statements are tailor-made to get libertarians to agree with them because they include both conservative “less government” sentiments and liberal “pro-choice” language. I’d also like to note that most of the so-seeming liberal social stances of libertarians stem from a desire for less government intervention more than they necessarily stem from actually agreeing with the particular social stance (gay marriage and abortion being prime examples), though this is not a hard-and-fast rule.
So, it’s pretty clear that the statistics generated by this obviously biased survey are about as useful as glossy magazine pages are for toilet paper — or as my education professor would say “Too slick to be of practical use”. Then why are so many articles (including ones from sources like HuffPo, MSNBC, and Yahoo News) referencing it as if it were actually true? Because it fits their narratives. And that’s a little bit sad because whatever your politics are, you should be able to get actual facts from your news. Sadly in these days of infotainment that no longer is the norm.
There is much talk lately about reforming science to improve its reproducibility (including the Reproducibility Initiative), and similar conversations about bringing transparency and accountability back to politics — but these problems are incredibly complicated and it’s going to be a while, if ever, before they are solved. Not that I’m advocating for giving up, by all means get involved with the process if you are passionate about it.
The best solution in the interim is to become an informed consumer of information. Identify sources of bias, question motives, think critically, evaluate the evidence. Or just say Frak it and go back to watching Big Bang Theory.
About the Author
Adrianne is a graduate student in biomedical science, technology enthusiast, former White House intern (not that kind though), and at blogs at Yeah, I made it. And her politics are complicated, just like her life.
