The show MUST go on — How Lockdown created a £19,000 AWS bill in just two hours!
It is said that ‘fortune favours the bold’. It does not however favour the foolhardy and sometimes the line between the two can be difficult to discern.
When the latest series of All Against One (AA1) aired in Norway this year in its usual position as the prime Saturday night show on the national broadcaster NRK, it immediately leapt into the ratings number one spot.
The format of the show sees a solo invited contestant playing against a formidable foe, in the guise of their fellow countrymen/women. While the contestant tries to guess the correct answer in the studio, the rest of the population (playing at home) guess via a mobile app. Their results are averaged, leaving two scores, the contestants and the country at large. Whichever is the closest answer to the question posed — wins.
№1 sounds fun but getting there was anything but.
Was it fate? And how it all started!
This particular session of AA1 started before lockdown, but as it hit the fourth show that situation changed, and the series was almost cancelled. AA1 is a LIVE TV show and so there were immediate concerns over how the broadcast could continue. However ironically, the format of the show lent itself to this challenge, albeit without the studio audience, and with the ‘celebrity’ element resorting to playing over a live video link.
With lockdown in full swing, audience numbers for the ‘play at home TV show’ unsurprisingly went up, with over a million (of the 5.5 million) Norwegians watching each Saturday night and at times over two hundred and fifty thousand playing by mobile phone.
As this particular weekend show approached, we were expecting an audience surge and so had ensured all stress testing had been based on very safe high-level estimates.
As time ran down, our problems started to mount.
However, on the day of the show, we hit a rather large, unexpected problem. With the predicted audience size, we had extrapolated that to be completely safe we should most likely run with two hundred PODS (servers) on the Amazon cloud for the live show itself.
As Saturday lunchtime arrived scale up began as usual, but then near disaster struck. Only moments into scale up we hit a hitherto unknown new limit of twenty servers, which Amazon had decided to set given the unprecedented demand of services going online. Amazon had taken the decision to reduce the server numbers allocated to each customer, and much like toilet roll rationing, we found ourselves unable to order what we needed with a potentially similar fate awaiting us.
With calm heads (?!) as the clock ticked down to the live show, we contacted Amazon’s Premium support who were (after a lot of patient explanations) very helpful in raising the limits. However, we then hit the next roadblock.
Amazon explained that due to demand, there were only five remaining machines of the type we required left in the region our build was in. This would have provided us with a grand total of twenty-five servers, around 175 short of what was required to run the show. With the clock ticking down we were in full scale stressed out mode and no amount of cajoling Amazon could change the number of free servers. They were not holding any back for their own use, they genuinely had sold out!
Sometimes a little gambling can save you!
It is at these times you get to see the metal of your team and when all seemed lost, our sysops team came up with a plan to rebuild the images on a different type of box. It turned out Amazon had plenty of their biggest most expensive boxes available and although we didn’t require the extra space it afforded us, it meant that by going big — we could go LIVE. It was a gamble we had to take.
We made a deal of £19K! What?!?
Within hours, the show went ahead and once again was faultless, no-one aware of the panic, consumption of coffee and loo roll that was going on behind the scenes. Success, but we knew it would come at a price. On Monday morning, when the dust had settled, we looked at the hosting bill — we had spent a staggering £19,000 in the matter of a few hours on AWS, compared to the few £K a typical show would have cost.
Having drawn straws, we then called the production company (Banijay), and with great trepidation we explained the situation to the Managing Director.
Expecting the worst, we readied ourselves for an uncomfortable ride. Whilst a £19k overspend was obviously a huge shock to him, he was in fact incredibly gracious.
But remember, fate decided to favour us?!
He told us “I told you to do your best to keep the show on air and you did. Without the mobile app there is no show. You made a judgement call and the show was a great success, smashing all records. I cannot say I am happy about the bill, but that’s not your fault. What I can say is that I am seriously happy you got everything to work so well yet again. Thank you guys, you saved the show!”.
