Facebook’s political ad spending numbers don’t add up
To understand ad spending on controversial tactics in this consequential election, we have no choice but rely on educated guesses
Facebook recently earned plaudits–from us and others–with an announcement that it would widen a ban on political ads after Election Day. Facebook is used by 7 in 10 U.S. adults, so their efforts to cut down on online misinformation in what may be a contested election are to be commended. However, we have found a critical flaw in one of their tools to combat misinformation, and Facebook has chosen not to fix it.
We have told Facebook that their political ad spend numbers don’t add up, and they acknowledge it. So now, more than two years since Facebook launched its voluntary disclosure of online political ads via the Facebook Ad Library, we still have to take some significant — if educated — guesses to know how much is actually spent on particular political ads or by specific political actors.
[B]y choosing not to fix critical flaws, [Facebook] makes it difficult for outside researchers to hold them accountable for what they promise to do.
How can this be? We’ll get into the weeds shortly, but the very fact that we have to make estimates at all is a big part of our point. With appropriate transparency tools, researchers and journalists like us would use these not just to inform the public with accurate trends in political advertising, such as political spending, but also to help the platform cut down on misinformation. But by choosing not to fix critical flaws, the platform makes it difficult for outside researchers to hold them accountable for what they promise to do.
We know this through experience. Our team at the Online Political Transparency Project has spent the past two and half years trying to learn everything we can about digital political advertising on Facebook and other online platforms, so we can develop better methods to detect misinformation, scams, and deceitful advertisers. Our home is the NYU Tandon School of Engineering, and we use traditional cybersecurity techniques to analyze social media.
Our public site Ad Observatory provides information for journalists to track trends on online Facebook spending. Ad Observer is a browser plug-in that allows users to volunteer data on what Facebook ads they are seeing, and why. Ad Observer is one of the sources for Ad Observatory, along with the Facebook Ad Library and Facebook’s Ad Library Reports.
Why does ad spending matter?
Facebook launched the Facebook Ad Library in 2018 to provide a searchable archive of ads on the platform. Facebook also makes this information available to researchers via an Application Programming Interface, or API.
After the 2018 U.S. midterm elections, our team developed methods to use the Facebook Ad Library information to detect undisclosed coordinated behavior, or advertisers misrepresenting themselves on Facebook. We also performed a rigorous security analysis of Facebook’s political ad library. In 2019, we attempted to put these methods to work in real time. That year, we monitored both the Canadian Parliamentary election and the UK snap election live, rather than doing an after-the-fact analysis as we had done in 2018.
A key lesson from all of this is that when trying to figure out what matters, the amount spent on an ad or a message is a really important signal, which would reveal trends in how political actors are deploying Facebook ads to influence voters. Ads can differ wildly in size, with some ads having less than $100 in spend, and others having more than $1,000,000. To make meaningful comparisons, we need to compare spending.
The importance of knowing spending on ads by day
We would like a dataset showing spend on particular Facebook ads by day because a daily per-ad spend estimate would let us answer important questions about how campaigns use online ads. For instance, we could accurately show shifts in spend over time as advertisers switched from running donation ads to running get out the vote ads! We could show which ad topics were trending in swing states! We could observe in real time the A-B testing process that advertisers use to decide which ad images and videos to put the most money behind!
Facebook does not report specific amounts spent on individual ads
But Facebook does not report spending on individual ads; rather, it provides a range of spending for each ad. This may make sense for the user who is searching the archive to see particular ads and doesn’t need that level of specificity. But it makes it impossible to compute how much an advertiser has spent on all the ads they’ve run with a certain image. Or topic. Or the many other ways we’d like to slice, dice, and analyze this information.
Incomplete information on ad run time
In the disclosures, Facebook indicates whether an ad is active, when it started, or is inactive. However, we do not know which days over this period the ad was active. If it goes inactive, we don’t necessarily know the date it stopped running.
Facebook has a way to solve these problems
Facebook actually has a solution on hand for this problem for researchers and other super users, without changing the display experience for those who simply want to browse the Facebook Ad Library. In addition to providing a searchable, visual way to browse ads via the library, Facebook publishes regular Ad Library Reports about how much each Facebook page has spent, along with disclosure of the entity that paid for the ad, or the “disclosure string.” Facebook also publishes how many ads these combinations had active each day.
Over the past six months or so, Facebook has reliably published this report daily, with a two or three-day delay for information about a particular date. When we saw this report was available daily, we figured we could cross-reference with the Facebook Ad Library information to compute how to more accurately estimate spend per ad per day.
How we calculate spending on ads per day
Here’s our methodology. (And a warning, this is super weedy.) First, we collect all ads, going as far back in history as possible, every day from the Facebook Ad Library API. In addition to other details about the ad, we record the ad’s start date and if it is currently active or not. From this, we are able to determine each ad’s effective start and stop date.
Second, we collect Facebook’s “lifelong” Ad Library Report every day, which lists, for each spender — that is, each combination of a Facebook page and “Paid for by” disclosure string — the amount spent by that spender over the history of the Ad Library. To get the most accurate estimate of how much was spent on a particular day by an advertiser, we subtract the amount spent up to the prior day from the amount spent up to the day for which we are estimating spend.
Finally, we distribute the amount spent for the most recent day for which we have an Ad Library Report to the ads we know were active on that day. We do this by dividing the amount spent by the number of active ads proportionally to those ads’ reported minimum spends, then adding that amount to the spend estimate for each of those active ads.
When we present spend data over time, we estimate the spend per day of an ad as the total estimated spend for the ad divided by the number of days the ad was active. This provides an estimate of how much was spent each day on each ad, which is why we can show you spend by topic or type, or over time for any subset of ads.
Facebook data is riddled with errors and inconsistencies
When we began to compare these two sources of information on Facebook political ads — the Facebook Ad Library and the Ad Library Reports — we found they were riddled with errors and inconsistencies:
- First, it’s important to know that Facebook publishes several reports every day for different time periods; one report covers all of history, another only the prior day as well as other specific time periods, including 7 days, 30 days, and 90 days.
- Sometimes, the lifetime spend reported in the Ad Library Report for an advertiser appears to decrease day-over-day. Often this negative delta is small, but it can be quite large. In one pointed example, we found that a page’s reported spend increased by an order of magnitude that then disappeared the next day.
- Sometimes, the difference in lifetime spend reported between two consecutive days can be quite different from the reported daily spend for that same period.
- Sometimes, spend and ad count both flicker up and down over the course of several days. So an advertiser on day one may have 100 ads reported, on day two might have 95 ads reported, and on day three might be back to having 100 ads reported. This means that on any given day, the most recent day’s data really can’t be trusted.
Facebook’s response to us
We’ve shared these problems directly with Facebook. We’ve been told that the root cause of these and other data inconsistencies in these reports is the fact that spend attributed to ads that ran on a given day can take up to five days to “settle.”
However, at any given time Facebook only makes available the most recent report– again, usually for a time period ending two days before the current day. They do not re-run reports after this five-day settlement period is complete or proactively make corrected reports available (Although they have offered to re-run specific reports when we’ve pointed out errors, to be fair). The only time you can be certain an advertiser’s reported spend in an Ad Library Report is correct is five days after that advertiser stops having any active ads.
The result: transparency theater
If your eyes aren’t glazed over at this point, you must also be a nerd like us. For most people, these details are downright snooze-worthy. But there’s a very practical result that has impact for everybody who uses Facebook or cares how political messaging on the platform is affecting the election: Despite promises for transparency, we instead get stuck with transparency theater.
Users shouldn’t have to pay attention to these kinds of details. We’re security researchers who focus on political ads. We take all of this much more seriously than most people do–because the public has a right to know how candidates and other political actors are spending and targeting voters via Facebook.
The problem is, Facebook is not providing real transparency about political ads. Facebook just says it does.
So where does that leave us? Shall we just throw up our hands and say nothing is knowable? Well, no. This isn’t good enough. Nowhere near good enough! We expect better from Facebook–and all the other online platforms that have been a major way political messaging has spread. Indeed, all online platforms should hew to the same standards of transparency and true transparency about political ads on their sites, including Google and other ad platforms. This isn’t an unsolved technical problem; it isn’t even a hard one. But in the face of incomplete, inaccurate information, we will continue to make the best estimate that is possible. You can read a concise summary of our methodology for spend estimation, including how we correct for one-day errors, here.
As a final note, as much as we have spent the last many paragraphs talking about Facebook and the many ways they need to improve their transparency efforts, it needs to be said that they have done more to make their ads transparent than any other major advertising platform. We do not even attempt to work with Google political ad data. This is because of myriad problems with how the company defines political ads, and therefore what it includes in disclosures, as well as the manner in which it provides information about ad creative content. But that is a story for another blog post.
Paul Duke, Jeremy Merrill, and Nancy Watzman contributed to this piece.
We will continue to write about technical problems with Facebook political ad disclosure. Check out Ad Observatory, a searchable site revealing trends in Facebook political advertising in the 2020 elections, and download the Ad Observer plug-in tool to safely volunteer information for researchers and journalists on what ads you are seeing on Facebook.
