Front Row Seat for S-Town
At PRX we’ve had the good fortune of working with the team at This American Life for a while, starting with our collaboration on mobile applications, then broadcast distribution when they became an independent public benefit corporation, and for the last few years in podcast ad serving (we aren’t involved in producing the shows, so don’t ask me when Serial season 3 is coming out). At each juncture, their popular and beloved programs have been inspiring to work with, but also challenging, especially technically. All that love can translate into strained servers.
No challenge had been bigger than when we took on ad serving for the Serial podcast, starting with season two’s launch. Season one was distributed the traditional way, with the now famous ads for “MailKimp” and other sponsors mixed into the episode audio, and the mp3 files served statically from a CDN. In the traditional model, the main bottleneck is that the CDN needs to scale to serve large volumes of requests and bandwidth, but of course, that’s what they’re made for.
For Serial season two, the plan was to get more advanced and use dynamic ad serving, “injecting” the ads into the episode audio. With dynamic serving, things get complex. Instead of sending download requests to the CDN immediately, our Dovetail media server has to specifically answer every request, figuring out which audio messages to stitch in based on the scheduled campaigns, and what we know about the listener from the download request, like their geolocation, device, and app they’re listening on. We joke that we’re making “artisanal audio”, with each file lovingly hand-crafted for the listener, but the truth behind the joke is that every request does get its own consideration, its own announcements, and its own sponsorship messages stitched in along with the show.
After getting Dovetail to work for Serial, I’d show off graphs of what it looked like when a new episode dropped and all the podcatcher apps decided to download it at once:
Whenever a new episode landed, in just a moment the already significant traffic for season one & two episodes would increase by six times. I love showing how Dovetail had been tuned to handle the load.
I thought that was pretty impressive, and that after Serial, Dovetail could handle just about anything. Then Seth at This American Life said they had a new show to put on Dovetail, and it would have a few differences. First, even though it was a Serial spin-off, it would be a whole new feed, so for better or worse it wouldn’t start with the existing Serial subscribers. Second, they wanted to have different ads in the audio from the start, but didn’t want them injected. They wanted the ads to be “baked in”, with two different versions of each episode, and each served half the time. Finally, (surprise!) all the episodes would drop at once, like a podcast version of a Netflix “binge-ready” show.
For this new show, which we now know as S-Town, having a new feed wasn’t a problem. If anything, I thought perhaps the system wouldn’t be hit as hard as it was by Serial’s two seasons of subscribers. That turned out to be correct but not meaningful; though starting from scratch, the initial burst of traffic reflected the enormous number of S-Town listeners who subscribed to the preview.
Serving up different but complete versions of a show was not a feature of Dovetail, and to some degree it still isn’t, but Dovetail is flexible and extensible. Given this challenge, we looked for a way to make the experiment work — I think I sent Seth about five different ideas to try, and luckily we had enough warning to figure out one that worked. Normally, the episode audio is the constant, and only the ads change. For S-Town, we created scripts to set it up in a unique way, reversing the usual role of the audio files; the ad would be the actual episode audio with ads baked in, and the episode audio would be a tiny silent audio file. The result was that half the time it served up the complete “Blue Apron” version of an episode, by stitching the Blue Apron episode audio as an “ad” onto the empty “episode” audio. The other half would be the “Squarespace” version, depending on whose turn it was as determined by the Dovetail traffic system. A neat trick, almost like A/B testing, and perhaps one we’ll figure out how to turn into a regular feature of Dovetail. For now it’s a successful experiment, and one we have the scripts and experience to build on.
The final difference was dropping all seven episodes at once. Our best guess was that this would create a spike greater than a single Serial episode, despite there being fewer initial subscribers. We more than doubled the cluster of Dovetail servers, and were prepared to expand further if needed. As it turns out, we didn’t need to at first, but the spike was just a bit higher than Serial’s. Below is a similar graph for S-Town lined up against that “impressive” Serial graph:
If a Serial episode was a mountain peak, then S-Town was the Himalayas; not only was the initial traffic roughly quadruple that of Serial’s, it just kept going, rising in the evening to a higher load than when the episodes first dropped. Serial episodes would often have a smaller secondary peak in the evening, but never higher, and never with such sustained traffic between the peaks. I’ve honestly never seen a download request pattern like what S-Town’s release produced.
Unlike when a single Serial episode would drop, where the sudden spike was followed by a downward slope as traffic abated, S-Town requests continued apace all day, and then day after day, with day two requests slightly fewer than one, day three practically identical to two, then only slightly decreasing in the days that followed.
While our estimate of servers was enough for the initial release, within the first day of S-Town, we upgraded Dovetail’s cluster to bigger machines to handle the unprecedented sustained load. We’ve made another round of improvements to caching and operations to keep it performing. After S-Town, we are that much more confident in our technology, both in new ways of using it, and under extreme load. Plus, the next time someone asks me what Dovetail can do, I have a new graph to show them.
