Snap’s bad data and our bad reading of it

Last week we wrote a story based on Snap’s latest political ad transparency data. The data was labelled inaccurately, so we drew the wrong conclusion from it.

Last week Snap released data about political ads that have run on its platform.

The data was wrong. It contained a column (labelled “Targeting Geo — Postal Code”) that appeared to include postcode districts that were being targeted with an ad campaign. In fact, the column included postcodes that were being excluded from the campaign.

Because the data was incorrect, we made an incorrect conclusion. Rather than the Conservatives targeting Boris Johnson’s Uxbridge seat to defend it, they were targeting everywhere else and excluding Uxbridge (likely to avoid needing to apportion any of that spending to his local campaign budget).

Bad data led us to a diametrically opposite conclusion to the one the correct data would have led us to. We’re obviously really sorry about that.

So here are the new facts:

1. The Conservatives ran some Snapchat ads, costing around £3,000.
2. To avoid them being seen as ‘local’ campaigning and having some of that budget apportioned to the campaign in Johnson’s seat, they targeted them outside of Uxbridge.
3. There’s no real ’story’ here in terms of the Uxbridge campaign (though our original piece included information about Labour’s campaign in the seat, along with some data from Facebook’s ad archive, both of which we believe to still be accurate).
4. Snap releases bad data and should work to ensure it’s reliable before releasing it. Other researchers should take note.
5. This was a new data source for us, and we should have found a better way to verify it before releasing our analysis.

Here’s what we’ve done:

1. Edited the original piece to make it clear that the analysis was based on bad data from Snap. Anyone visiting that link will now find the correction and not the original story.
2. Removed the original tweets and Facebook posts about the piece.
3. Notified the media who picked up on it that the data supplied by Snap was faulty and that future pieces based on their data should be treated carefully.

This is deeply frustrating for us and risks people’s trust in our project. We’ve worked hard on the problem of targeted political advertising for two and a half years and take great pride in our work.

As a small organisation working to increase ad transparency during elections, we expect big companies, with large resources, to check that the data they release is worth releasing. We work in an emerging field, where many researchers, like us, are making a good faith attempt to ensure that modern political campaigns are properly explained and held to account.

Until the standards for political ad transparency are mandated in law, with suitable penalties for non-compliance exist, these types of mistake will continue to happen. In future, we’ll try and make less of them.