Süddeutsche Zeitung is improving the way media reports on political polls
The data team visualises polling uncertainty for clearer campaign reporting
Polling is ubiquitous in political campaigns and it influences everything from ad buying to voter turnout. Yet polling methodology and reporting have been the subjects of renewed scrutiny since Brexit and US election in 2016.
In anticipation of the 2017 German parliamentary elections, Süddeutsche Zeitung (SZ), the Munich-based newsroom that originated the biggest leaks of the past couple years, tackled this challenge. They isolated two polling issues to solve in their reporting:
- Polling methodology is flawed. Sample groups are not representative, people may be untruthful producing skewed results.
- Polling institutes and news media don’t communicate well on these ambiguities.
This was the conundrum facing the team from SZ as they headed into the Editors Lab that they hosted in October 2016. Their headquarters in Munich was a the backdrop for newsrooms from around the region such as Spiegel Online, Le Monde, OCCRP, Neue Zürcher Zeitung, and Der Standard.
Less-than-perfect polling
To address flaws in polling methodology, they decided that more data would result in higher certainty. ‘Every polling institute has its own way of conducting its survey: How big is the sample size? How do they weigh different demographics? How do they treat undecided voters or non-voters? Therefore every survey is wrong in its own way’, said the team. Still, the SZ hackers recognised the value in aggregating as many sources as possible. They collected results from seven polling institutions and as a whole, ‘They provide valid information about potential voting patterns of the electorate. Therefore a smarter way of reporting about opinion polls is to get as much data as possible’.
The team can measure and visualise this statistical uncertainty but polls can also be affected by ‘systematic’ uncertainty. These polling errors can be caused by voters flip-flopping or failing to be honest with survey takers. Honesty turned out to be a major polling flaw in the US presidential election with many of those polled lying about their plans to vote for Trump, as reported by data journalist at the Guardian Mona Chalabi. She took these polling flaws into account and predicted a Trump victory three months before the US election in 2016.
Communicating doubt
Now to address the second issue of reporting and visualising the inherent uncertainty of polling results. The media typically communicates only the mean value of a given poll result. What we don’t usually see is that the mean value is contained within a margin of error. According to the team, ‘a better approach would be to publish the mean along with a confidence level. That can be interpreted as a 95 percent chance that the party’s result will be between 10 and 15 percent. SZ developed a way to visualise this confidence interval. They show the mean value within a range of polling data, rather than just the mean. This method is much clearer about this lack of clarity.
The new charts were deployed for the election campaign coverage beginning in spring 2017. SZ were transparent with their users about this new methodology, publishing an article explaining their process as well as open sourcing the project.
The published versions were the result of much user testing. ‘Again and again we showed the charts to readers and colleagues and asked for feedback, because we knew that it is harder to [understand] a rather uncommon visualisation. For example, one of the conclusions was to include smaller icons in the caption — sparklines —to make it as easy as possible to understand,’ said editor Katharina Brunner. ‘Over time we created several versions in order to optimise correctness, understandability, and design’.
Regarding the choice to keep the centre line within the blocks of colour representing the polling range, Brunner explained, ‘It was a sacrifice for the sake of faster and easier understandability’. She added that it was a decision that made the charts clearer about the parties that get smaller percentages of the vote, whose intervals overlap quite a bit. ‘Our user research showed, that it helps to keep track a single party’.
They plans to continue to work with her team to fine-tune and reuse the charts for the Bavarian elections this October. Although the flaws in polling methods persist, SZ is well on their way to making sure the public has the clearest picture possible as they head to the voting booth.
