The Ad Tech Landscape Today: Existing Problems As We See Them

Hello friends! We are software developers at ConsenSys and MetaX. You may have heard that we are collaborating to solve the very expensive problem of fraud in the web advertising industry. We’ll soon be releasing a technical whitepaper describing our ambitious plan, called AdChain, to resolve the coordination and incentive problems at the root of ad fraud. In advance of that, we want to share our view on the existing landscape as we see it. Perhaps you can think about how you would use a blockchain to solve these problems in advance of our whitepaper release, so that you will be primed to discuss these issues with us when we issue our proposal.

Cheers!

Mike, Ameen, James, Edwin & Miguel

***

Terminology

Ads served by programmatic marketing campaigns comprise the vast majority of ads users see while browsing the web. Consequently, programmatic ad buying comprises the vast majority of revenue in the web advertising ecosystem. In programmatic ad buying a publisher such as the New York Times makes advertising space on a web page available for purchase and serves that space to users without pre-negotiating an ad placement with an advertiser. This is as opposed to the premium direct buy market, wherein ad placement is pre-negotiated.

Internet advertising can be abstracted as interactions between three parties. They are:

• The User who browses/navigates the internet and various desktop and mobile web pages
• The Publisher who owns and controls the web pages and generates content that users consume
• The Advertisers who desire to get the attention of users and pay publishers to place ads on their web pages

How Web Ads Are Served

The creation and delivery of digital ads is an involved and detailed process involving many parties. In order for brands and advertisers to negotiate and execute advertising deals, or ad campaigns, there are several intermediaries involved, many of which provide overlapping services. Advertisers often use ad agencies to create marketing campaigns and campaign assets called creatives. These ad campaign creatives are served from Demand-Side Platforms, or DSPs. On the supply-side of the industry, where publishers sit, there are SSPs, or Supply-Side Platforms. A common way to buy and sell ads is through an exchange, which is an aggregated marketplace of DSPs and SSPs. This is where things get complicated.

Ad networks aggregate publishers and work with SSPs to solicit ad campaigns in exchanges and with large DSPs directly. Publishers may work with ad networks, SSPs, exchanges, DSPs or directly with brands and advertisers. Publishers often work with ad tech companies across all categories in the ecosystem simultaneously in the effort to monetize 100% of their advertising inventory. Conversely, advertisers and agencies typically work with several different exchanges, SSPs and publishers to reach their target demographic at scale.

Fig. 1 Programmatic ad paths

As an example and from the perspective of a unique creative element in an ad campaign, an ad’s journey to a user’s browser begins when the New York Times embeds a video player in their homepage. When the user loads the page the video player itself, from the user’s browser, issues a bid request to any number of ad exchanges over a protocol called RTB (Real-Time Bidding). Advertisers purchase that space through a DSP, trade desk or exchange and the advertiser who wins the bid gets their video ad played. In the happy case, the New York Times, the exchange and a number of other firms in that ad’s supply chain all get paid.

This is rather complex on its own. It gets significantly more complex when one considers all the other firms that may be present in a supply chain. The web advertising technology stack as it exists today grew in the very dawn of the web itself, and a major problem in ad tech is the inability for parties in a supply chain to share information before the close of 30 to 60 day settlement cycles defined in paper contracts.

Returning to the earlier example, when a web video ad is served on the New York Times front page, a large number of tracking beacons are served along with it; in general one tracking beacon per supply-chain entity. These tracking beacons assess, among other things, whether an ad was viewable and whether its viewability was sufficient to log an impression. Payments in web ad supply chains generally happen per 1000 impressions, which the advertising industry calls a CPM (cost per mille, where mille is French for thousand).

In some instance, the New York Times’ tracker may assess that an ad was viewed in its entirety, whereas the advertiser’s safety vendor (another common supply chain entity) may report that while the video did play, it was not in an active tab and so doesn’t count as an impression. Or the safety vendor may suspect a bot viewed the video; fraud of various types is a $10 billion+ problem in ad tech annually.

Because of the opacity of web ad supply chains, at the end of a settlement cycle the New York Times may expect to collect 100,000 impressions worth of revenue and are surprised to learn the advertiser is only willing to pay for 5,000 impressions on the basis of data from their safety vendor. This dispute can only be resolved following a costly and time-consuming negotiation. Nobody makes as much money as they believe they should have and everyone walks away feeling suspicious of everyone else. This is extremely common in ad tech, to the extent that there are companies which purchase ad contracts from publishers and other supply chain entities at a discount on their face value and bet on their ability to negotiate a profitable outcome in settlement! The supply chain entities settle for being paid some fraction of what they believe they are truly owed just to ensure proper cash flow to cover current liabilities.

Web Ad Supply Chains

When we refer to a web advertisement’s supply chain we are referring to the multifarious parties involved in serving that advertisement to an end-user. Between an advertiser (Coca-Cola) and a publisher (The New York Times) can stand upwards of ten firms, and generally at least six. Some of these firms are employed by the advertiser, a smaller number by the publisher, and several may be employed by other firms in the supply chain itself. All of these firms take a cut of revenue whenever payment flows from an advertiser to a publisher.

Of the six “core” parties present in most web ad supply chains, the advertiser and publisher constitute two. A DSP and a safety vendor are employed directly by the advertiser, and the DSP will have arrangements with any number of ad exchanges. The exchange through which an ad is eventually placed becomes a part of that ad’s supply chain. A publisher will employ an SSP and may also employ its own safety vendor. What do these firms do?

Demand-Side Platforms

The core competency of Coca-Cola’s marketing agency is in the production of compelling creative material which convinces consumers to Drink More Coke. In 1966 it was within their competency to pick up a phone and negotiate ad placements in newspapers, magazines and on television. The 21st century programmatic ad buying marketplace is much too complex for a creative agency to effectively place ads in; instead they contract with a specialist firm which provides a DSP. DSPs provide an interface for advertisers to upload the creative content they produce, specify how that creative should be targeted and a budget for placing the creative. DSPs have relationships with multiple ad exchanges so they can place ads more effectively across the largest possible number of publishers.

Safety Vendors

As mentioned in the introductory section, How Web Ads Are Served, there is often disagreement between supply chain entities about what count as “payable” impressions when ads are served. In fact, because the supply chain only gets paid per 1000 impressions, every actor in the supply chain aside from the advertiser themselves are incentivized to create as many “impressions” as possible, and to do so by any means necessary. This may mean exchanges serve ads to “off-brand” publishers and bot farms, or supply chain entities simply inflate impression counts arbitrarily. Even when fraud is not at play, technical errors and genuine differences of opinion expressed in a tracker’s business logic regarding what comprises viewability can lead to significant deviations in impressions reported by various parties in a supply chain.

Because advertisers themselves are not sophisticated in auditing modern web ad supply chains, they typically employ a safety vendor to provide expert analysis of viewability and potential fraud in their supply chains. Safety vendors have strange incentives: as they are paid on a CPM basis, in a sense they are only incentivized to reduce fraud as much as their competitors do and no more. They depend on the continued existence of fraud for their business to survive.

Exchanges

Ad exchanges match bid requests on ad space placed by publishers to bids on ad space placed by advertisers. The utility of an exchange in web-scale advertising is to rationally allocate available ad space to the advertiser to whom it is most valuable. Because there are many ad exchanges, the supply and demand sides are both fragmented and the rational allocation of available ad space is impeded. The proliferation of exchanges also helps to keep DSPs and SSPs in business by creating yet another barrier to entry for advertisers who might like to place their ads directly. Furthermore, ad exchanges operate with complete opacity and neither publishers nor advertisers are able to audit whether an exchange manipulates its order book to optimize its own fee extraction. There is evidence that such practices are in fact commonplace.

Supply-Side Platforms

Just as advertisers’ core competencies are not in the actual placement of ads, publishers’ core competencies are not in selling ad space. The New York Times is a journalistic organization, not a sophisticated exchange arbitraging outfit. Publishers therefore contract with SSPs for similar reasons to those advertisers contract with DSPs. SSPs have relationships with multiple exchanges so they can fill ad space with the highest paying ads.

Vendor Lock-In in Web Ad Supply Chains

A fundamental inefficiency of web ad supply chains relative to how they could function in the Internet age is that they do not allow for granular composition of ideal supply chains on an ad-hoc, per-impression basis. Advertisers are bound by ongoing contractual relationship with their DSP, and if they want to work with a different DSP with a different expertise for a particular campaign they will need a second contract. Safety vendors lock advertisers into contractual obligations as well, so an advertiser who wants to work with a safety vendor specializing in click-fraud for one campaign and bot farming for another will need long-lived contractual relationships with both. The frictional costs of changing vendors prevents the composition of ideal supply chains formed on the basis of what a particular ad campaign, or even a particular creative element in a campaign, needs. Supply chains are sticky in a way that benefits the supply chain itself, rather than advertisers or publishers.

Ad Exchanges: Zero Transparency

Ad exchanges have great latitude to misbehave in modern web ad supply chains. Web ad exchanges do not actually expose the same degree of information about the state of an ad market which a financial exchange for securities or commodities might. Data pertaining to exchange volume and average sale prices, for example, is generally unavailable. In order to develop an image for the state of a market, buyers on the demand side need to probe the exchange with high-frequency bid responses for ads they do not intend to actually serve. This inefficient practice cannot perfectly expose the state of a market, as ad exchanges may push floor prices upwards arbitrarily and undetectably. Ad exchanges may alter bid requests arbitrarily to increase the fees they are able to collect; lower-value javascript inventory may be sold as high-value video inventory, and buyers have no way to detect at bid-time that they are buying subpar space.

While every hop in a web ad supply chain is opaque, exchanges are especially privileged because of their position at the intersection of supply and demand, and their ability to mark up the value of available supply to buyers arbitrarily.

Lost Opportunities in Data Wholesaling

End-users are targeted and tracked without their knowledge or consent by a wide array of ad tech companies who sell audience data to advertisers. Companies track users with proprietary tracking methods that deploy “tracking pixels”, or beacons, directly to web pages, which negatively impacts user experience by increasing page load times and hurting performance. The fact that multiple private companies do this work results in fragmented data trails and decayed ease of access where two measurements of the same thing may have different names. “First party” data ostensibly owned by publishers is easily stolen and repackaged for sale by third parties, and theft of this nature is impossible to detect. There exists no means for data purchasers to subscribe to data feeds that aggregate supply chains industry-wide. There exists no means for users to collect compensation for the data collected about them or participate in the upside of its analysis.

Supply Chain Opacity and Clawbacks

The digital advertising ecosystem lacks common systems for coordinating and verifying data about viewability and fraud. This inability to coordinate stems equally from misaligned business incentives and the absence of a coherent proposal to furnish such a system in a way which does not add additional overhead to supply chains. Digital advertising supply chains consists of self-interested parties that rely on shared protocols for ad delivery (VAST, VPAID, RTB, MRAID), but depend on proprietary partner APIs and closed technology stacks for performance tracking. When parties need to reconcile data, multiple in-house and 3rd party measurement tools are used for cross-ecosystem reporting. With multiple parties reporting on a single advertising impression event, discrepancies are common. In fact, one of the leading programmatic ad tech companies found a 40% discrepancy in ads that were being called and ads that were being served. The lack of interoperable standards for performance tracking not only causes market inefficiencies, but creates opportunities for bad actors to exploit the lack of transparency.

An interesting example of how this problem manifests in the real-world is in the degree of conflicting information provided by safety vendors. Proprietary algorithms filter and catalog what is believed to be fraudulent or ineligible inventory, and the assessments made can change daily. A supply-side ad operations manager may see one buyer opting out of buying placements on a particular domain while the very same domain could be top-performing in the assessment of a competing buyer using a different safety vendor. What is troubling for an ad operations manager witnessing such a phenomenon is the fact that advertisers have the ability to claw back payment on a placement months after the inventory was purchased; this leaves the supplier operating in a constant state of uncertainty.

When a claw back does occur, potentially months after the fraudulent impression event was recorded, the supplier may discover that many popular ad servers do not store data longer than the settlement period of the campaign. A supplier who has been hammered with a claw back trying to locate the inventory provider often finds out too late that this has become impossible. The downstream provider may not even be in business anymore or may be doing business under a new name, enabling them to relinquish responsibility on previously sold inventory. This is akin to a manufacturer selling a faulty product and then, months later as complaints roll in and the lawsuits begin to pile up, being found to have gone out of business with all of its executives having left the country.

Fraud: The $10+ Billion Problem

The online advertising industry has evolved to become a tangled mess of algorithms, virtual real-time platforms and exchanges all competing for consumers’ fragmented attention. Long gone are the days where an advertising executive could pick up the paper and see the full spread ad they purchased the week before on page 12. There is no more paper trail. Old school broadcast media knowledge and strategy has not translated to the digital space. We now live in a complex automated world where online advertising is programmatic and it is cheaper than ever to evaluate the real-estate on publishers’ websites. As a result, there has been a dramatic increase in ad inventory and there is economic incentive to place an increasing number of less valuable ads on lower-end corners of the internet. Ad tech companies and intermediaries sprout up like mushrooms, entering into the unregulated market looking to capitalize on ever growing advertiser demand. Online advertising fraud has exploded in this environment.

Ad fraud on the supply side is realized primarily in the form of botnets and click fraud. The fraudster brings a website or number of websites online and sells ad inventory on them. The fraudster then points a botnet to the fraudulent website. Botnet behavior is increasingly sophisticated and difficult to detect; bot farmers are intelligent about maximizing the apparent value of their non-human audience members by having them load up carts on shopping sites and browsing for luxury cars. Because publishers and advertisers are so far removed from one another by aggregators and exchanges, it is nearly impossible to detect bot fraud proactively.

On the demand side, fraud is primarily perpetrated using malware injection. An advertiser may serve a legitimate looking ad which installs malware on clickthrough by exploiting vulnerabilities in frameworks like Flash and Silverlight. By exploiting the same ostensibly legitimate strategies as other advertisers in clawing back payments, malware can be spread through ad networks very cheaply. Because the ad market is completely unregulated, affected users have no recourse.

Vertical Integration: A Solution With Side-Effects

In 2017 a duopoly captures upwards of 60% of the programmatic ad market’s total revenue. This duopoly’s large audience, combined with their deep targeting data about that audience is an extremely valuable commodity. The ecosystem these giants of ad tech play in is largely vertically-integrated and comprises more than just the supply side: they own their entire supply chains, and the only means to serve ads to their audiences require passing through that infrastructure. These extremely hygienic supply chains, combined with the duopoly’s world-class audience targeting data allow the duopoly to capture even more demand than they otherwise would strictly on the basis of their high quality audiences; in serving ads through the duopoly, an advertiser has extremely high confidence that they will not be paying out to botnets or participating in supply chains that serve malware to end users.

In a sense, vertical integration is a solution to many problems in ad tech. It also may be a trap, and the social and political repercussions of the dominance of a duopoly are disconcerting. For better or worse, advertising is media and media is culture. A duopolistic elite guarding an important path to information dissemination on the Internet appears at first blush to be a Very Bad Thing for open societies. Firms which comprise the market outside of the duopoly should be highly incentivized to work together to break the duopoly’s grip by improving the quality of their own offerings, lest they be destroyed by the duopoly’s economy of scale. To do so will require resolving the coordination and opacity failures in existing RTB supply chains.

The online advertising industry outside of the duopoly is free, open and increasingly peer-to-peer thanks to emerging header bidding technology. It is suffering, however, under the burdens of mistrust, fraud, supply chain opacity and absent accountability. The infrastructure of advertising on the open Web needs a coordinating mechanism to unite it and address these issues.

In future blog posts and our forthcoming whitepaper, we will describe how AdChain can serve as that coordinating mechanism. Stay tuned!