Antiquated technology is hurting the real-time bidding industry. It is about time, all of that changed.
I am sure that the above title raised a lot of eyeballs since RTB spend is increasing consistently each year.
However, being someone who operates a RTB platform, I can make a few assertions (about the industry) with a great degree of confidence:
- The Real-time bidding (RTB) protocol is bloated and hurting (to the point of killing) the ad-tech industry
- Large players invest in extremely large amount of hardware to offset the complexity introduced by the RTB spec; This together with other expenditure explains why many of them are unable to reach profitability even after going public.
- Small players can’t compete with large players effectively because the amount of infrastructure required is beyond what they can afford, raising the barrier to entry in this industry.
- All of the above is hurting competitiveness on the whole and limiting advertiser choices.
RTB is not exactly cutting edge
Real-time Bidding (RTB) operates as follows: Publishers send requests which are initiated through a browser as HTTP requests and expect responses through HTTP as well. To understand each other’s requests, servers exchange JSON data between one another that complies to the RTB Specification.
The RTB spec mandates more than one request / response cycle even just to serve one ad. Multiply that by about 50 billion ads that are approximately served each day and you get the number of HTTP requests that the RTB industry uses to just one platform. Even a blank HTTP requests is several tens of bytes. A valid RTB request could be up to a kilobyte in size.
That equals about
465 terra bytes of data
moving between 100 advertising platforms across the world, each day.
Yes, you read that right..the ad tech industry transports tremendous amounts of data over HTTP protocol, through billions of connections instead of using a better or more efficient means of transport.
Ancient Approach; Bloated technology
Even though all communication in RTB happens between exchanges, supply platforms and DSPs, the interaction between these various server to server processes happen on very a very ancient technology —Real-time Bidding Spec (RTB) over HTTP. That approach maybe completely unnecessary.
Hyper Text Transfer Protocol (HTTP) was built as an interface between a browser and a server. Because the originator of an ad request is the browser, the ad industry got stuck using it even for server to server communication. Using HTTP for server to server communication is the equivalent of using a bus exclusively booked for each passenger to go back home. A bus was built for mass transit but using it for a single passenger takes up unnecessary road space, delays (busses are slower) and leads to higher costs.
Likewise, HTTP as a transport is slow (it is text based), it uses up more compute and network resources and is needlessly bloated. It is especially unnecessary when two servers are communicating to each other.
Even the RTB Spec is poorly designed.
The RTB spec aims to be the lingua franca between buyers and sellers. It is a clean verbose protocol that is a delight to read by us humans. Unfortunately, it is too verbose and can get too heavy without compression. Consider the following json ad sample given on page 39 of the RTB Spec. It weights 1.0KB in size, without the data required by the HTTP transport. That is just for one part of the RTB conversation.
The Impact on the Advertising Industry
The bad design of RTB and the use of HTTP in general has created several issues in the Ad tech industry. They are given in no particular order:
Excessive costs thanks to bloated server infrastructure requirements. HTTP is not a light weight protocol. In fact it is extremely bloated. And RTB is a text based protocol that operates atop HTTP. HTTP requires massive infrastructure deployed at scale to be supported. Why? Because HTTP is stateless ( and requires plenty of headers to be passed back and forth in a connection), it is also mostly connection less. Some folks might argue that using keep alive helps. Yes, keep alive helps in not re-creating connections. However each connection is exclusively used and there is way too much data being passed back and forth.
Delayed responses leading to no bids — Ad exchanges that respond a few milliseconds later than competing exchanges resulting in not even being able to bid for the user, let alone being able to show ads to the user.
Loss of revenue and ROI due to increased latencies — Network latency is one of the most serious issues affecting ad serving. More often than not, increased latencies lead to certain platforms claiming to have delivered the ad impressions between publishers completely ignoring them (by not even showing the ad impressions), leading to loss of revenue to the publisher and loss of ROI for the advertiser.
Non Viewable Ads — Viewability is the up and coming metric that has become increasingly common amongst brand advertisers. Fewer ads are shown and therefore viewability as a metric gets reduced for many of those exchanges which cannot respond on time.
Solutions
All is not lost for the ad tech industry. Given that most emergening platforms also need to handle really large scales (10 + billion minimum upto 50 billion ad requests, each day)…the systems need to be rethought ground up with focus on latencies, speeds and efficiency. I have a few suggestions that I think will make a lot of difference:
- Move to SPDY / HTTP 2 for transport — SPDY’s and HTTP2.0's multi-plexing, compression and prioritization can reduce the overall infrastructure required for RTB infrastructure to operate, reducing costs and improving latencies.
- OR, Move to TCP/IP for transport. It is an open secret that TCP/IP itself could be used to support millions of active concurrent connections on commodity servers. TCP offers a much faster method of data delivery between different RTB partners, increasing speed and reliability of RTB messages.
- Follow Facebook’s lead and move to FB Thrift instead of a plain text RTB 2.3 spec (https://github.com/facebook/fbthrift). Thrift along with a better transport mechanism can do wonders for RTB and it would need elimination of the Real-time Bidding protocol in its current shape and form.
- Alternatively, abandon the current RTB spec and rewrite it in the form of a really light weight protocol with strictly TCP only for transport between servers.
Thoughts, comments?
Get in touch with me at azifali@gmail.com or asif@reducedata.com
