Non-probability polls and their usefulness for news organizations
Are statistics from non-probability polls an accurate representation of public opinion?
The answer to this question is, “it depends”.
First let’s define what non-probability polls actually are. Unlike probability polls, in a non-probability poll participants are not selected at random but are included by other means, such as volunteering. As a result, the person’s chance of being in the sample is unknown, according to the American Association for Public Opinion Research. A popular example of this is an opt-in survey, where a user accepts an invitation to complete a survey offered to all site visitors.
There are now strong arguments that opt-in online surveys can be regarded a valid polling method — especially for news organizations. In fact, the 2018 AP Stylebook includes a new chapter on polls and surveys that supports this claim. AP says that “some cutting-edge methodologies that incorporate opt-in online surveys may, after thorough review, be suitable for publication.”
If an organization is looking at non-probability polls as one of many data inputs, they may be useful as a real-time temperature test of what people think.
There seems to be a perpetual wrestling match between traditionalists and innovation — those pushing the boundaries to find better ways to capture and measure public opinion.
So what does the accuracy of non-probably polls depend on?
A few things:
- How well the poll is distributed to enable a diverse range of viewpoints.
- Whether the methodology has been explained and reveals numbers.
- Whether bias has been accounted for in the questionnaire.
- If there is a set population and filtering of results of those who do not qualify.
- And whether or not you have explained that the percentage numbers are only representative of those who chose to participate.
Peter Kruit, data science advisor to Poll Town, confirmed our polls are done using non-probability sampling and suggested ways to improve the data accuracy, including:
- Limiting the number of choices participants have in front of them, and
- Designing for high completion rates (so complete data sets can be reported on).
Bitsy Bentley, a seasoned data practitioner from New York, says that no data is perfect and every system has its downfalls.
She says, “As long as we disclose the compromises we are making in order to get the widest engagement and participation, then we can stand behind the results we are releasing”. Bentley is ambivalent about calculations like the “margin of error” and “statistical significance”, instead preferring to talk about hypothesis testing and measuring significance in relation to what were the expected findings versus the actual findings.
Bentley could see the potential of Poll Town results being used to validate or dispel findings from other feedback mechanisms being used by civic leaders and decision-makers.
Joy Marie Sever, who has written widely about polling insights, emphasizes the importance of question design. She says that polling organizations need to be exceptionally careful with wording so that results are not skewed.
Yet Sever says to remember that being representative is hard. Polls can build credibility by being able to reach consistently large numbers, and with the right level of coverage, polls can be more representative.
When reporting results, Sever recommends that polling organizations be clear about who the result represents, such as “X% of the respondents said Y.”
Peter Young, Poll Town’s ongoing advisor, is a journalism lecturer who has the unique claim of running the first-ever Internet poll. Young urges polling organizations not to worry about sampling as much as the margin of error. “If the margin of error is above 3.5 percent, the data may not be helpful,” he says.
Peter explained there are two types of survey professionals. Firstly, the Nielsens of the world, who are obsessive about their sampling. Then there’s the other 98 percent of the population, who conduct surveys “of convenience”. Most news outlets cover the latter, he says.
“In our fast-paced digital lives, convenience seems to be king, however, no-one believes a Twitter poll really means anything”, says Poll Town founder and CEO, Keren Flavell. “We’ve kept figuring out how to broaden the accessibility of the poll much wider than the social media channels alone, while building in safeguards to limit gaming of the poll to skew the numbers.”
“The quest for data accuracy is on-going and important, so we can distribute meaningful poll results that show community sentiment on important issues”, says Flavell.
Ready to engage your audience with online polling? Visit Poll Town to start your own poll and see reports from previous polls.
