How To Spot a Fake News Poll

George Khalaf
Aug 24, 2017 · 3 min read

With the number of polls increasing dramatically in the last couple years, it is becoming harder to decipher what is accurate and what is not. It becomes increasingly harder when many media outlets publish polling numbers that are questionable at best simply because the results are “news worthy”. Inspired by Harry Enten at FiveThirtyEight who wrote “How to Avoid Falling for a Fake Poll”, we wanted to put together a more Arizona centric version. As a firm who does quite a bit of polling work in Arizona, here are some things we have learned to do, and more importantly avoid, when it comes to polling.

1) Polling “likely voters” will yield the most instructive results.

If your objective is to make ANY implications on future elections or legislation, you MUST talk to people who are most likely to vote. Simply talking to heads of households or registered voters won’t make for accurate election analysis/predictions. In Arizona, Democrats and Republicans are within 1% or 2% when it comes to raw registration numbers, but Republicans (in a typical non-Presidential election) will gain a little more than a 10% advantage on Democrats when it comes to actual voter turnout. Even more glaring, people over the age of 65 represent only 13.8% of the population (according to 2010 Census), while they typically represent somewhere between 35% and 40% of actual voter turnout.

2) Live caller surveys will yield the most trusted results.

We learned this lesson the hard way. We ran an automated-only survey for a client and got results that we just knew could not be accurate. Once we ran the survey using live calls, the results changed by 20%! Because polling firms are legally not allowed to reach people via cell phone using an automated system, it is harder to talk to certain groups of people less likely to have landline phones (younger people and Hispanics, in particular). There is a place for automated surveys (we use them a fair amount) but precautions, such as a higher than average sample size, need to be taken. Many pollsters are attempting to conduct more survey work online as the percentage of people with landlines and reachable cell phones shrinks, but we aren’t there yet. Online survey work should, by and large, be taken with a grain of salt unless a lot of due diligence is done.

3) Using a large enough sample size is critical.

With a state the size of ours, any statewide poll with less than 550 completed surveys runs a much higher risk of not being truly representative. We aim for between 550 and 600, and sometimes as high as 700, depending on the type and intention of the survey. Congressionally, you should be around 450 to 500, while you should be around 400 to 450 on a legislative level or for your average Arizona city.

4) Transparency, Transparency, Transparency.

There is nothing more important and more widely agreed upon in the polling community than the notion of full transparency. This means Publishing your results, demographic breakdowns, crosstabs and even, in some cases, raw results. Full transparency is the only way to truly tell a good poll from a bad one. At a bare minimum, if a pollster doesn’t publish their demographic breakdowns, then the results need to be treated with caution. Opening yourself up to criticism (we have had our fair share, especially early on) is the best way to avoid them in the future.

I know the polling industry has taken a lot of flack because of “inaccurately predicting” past election results and there is no doubt that polling will continue to be imperfect. However, limiting the amount of error is a good step forward. Even more important than pollsters accidently or intentionally producing fake numbers are folks in the media being far more careful with what is published. If bad polls aren’t given the time of day, they are less likely to be repeated.

George Khalaf is President of Data Orbital, an Arizona based political consulting, data analytics and polling firm.

)
George Khalaf

Written by

Political Operative. Pollster. President & Data Strategist at @Data_Orbital. Catholic. Husband. Father.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade