1 Tbps mafia defeated by bureaucracy in government tender

Our goal for 2016 was to reach a 1 Tbps bandwidth from CDN77.com servers. At the beginning we were at about 300 Gbps.

Most people were slightly skeptical yet I was confident.

To remind the team of this goal, I named our network team a 1 Tbps mafia.

I even put a sign on their door. Lo and behold, this simple sign actually helped.

The last few weeks before Christmas have been like an endless adrenaline party. Our network recently surpassed a 1.2Tbps peak traffic. The traffic now flows from approximately 5,000 Ethernet ports, 10Gbps each. The network department is having the prime time of their life and we‘re finishing with a new cool feature for live streaming.

1 Tbps traffic — 10/2016, now we push over 1.2 Tbps to the internet

Why are we building a feature for live streaming?

Well, a considerable portion of the traffic already consists of live streaming and the increasing trend is obvious. Furthermore, we came across an interesting opportunity roughly 6 months ago during which we got the product idea for WebRTC P2P streaming. And yes, 6 months is a long time but hold tight. Everything is going to make sense. Eventually.

First things first, part of our development team is located in Prague. It’s rather a small market for business, but a great source of talent for a tech company.

We were not the first company to figure that out, think of Avast, AVG (recently merged with the former), GoodData, Merck R&D, Apiary and many more. It’s a rapidly growing area with a strong tech community and a hidden treasure when you seek for talented people.

Now, Ceska Televize (Czech Television, CT) launched a new tender for online broadcasting and on-demand TV shows in June 2016. Not since few days before the tender deadline had we any idea about it.

Ceska Televize is the largest national TV on the market with around 10 000 000 people.

Our network operations team @ CDN77.com

It sounded like an interesting deal. After all, it’s a national television. As hard as their employees try, however, it’s still a public institution and bureaucracy is rooted deep. CT’s technology stack isn’t really up-to-date — Flash player, architecture that doesn’t spare servers and gives them rather a proper load. Put simply, year 2012, nothing special really. What a room for improvement.

The tender documentation was thick. Like real thick. Tens of pages of dense text of what needed to be done. Single-handedly, the television just hands you over a few cables. The rest is on you — transport to your datacenter, live recording of all channels, transcoding to several formats, Internet distribution, unpredictable peaks scaling, HTML5 / JS player, video-editing dashboard for editors. Besides that, a few more details such as 4K support, licensed Dolby Digital Plus sound or Widevine support.

In fact, the whole contract was ensured by a decent penalty table for any vendor’s misconduct. Understandable.

Anyway we wanted to try it. We made a binding offer within 100s hours.

The assignment was clear. Whoever complies with all formal and technical criteria and makes the lowest bid, they win. Fair deal you’d say.

We fulfilled all the formal requirements and offered the lowest price. For comparison, the runner-up company offered a higher price by nearly $150,000. It’s a significant difference in the context of an estimated million dollar deal.

To showcase what we’re capable of, we created a technology demo. A functioning prototype built in 14 days. The prototype was capable to receive a video in a diverse range of obscure formats, encode it live to 10 different formats and output the stream in HLS, MPEG-DASH and other formats for HTML5 player. Meanwhile, another team started working on the live WebRTC P2P video distribution for networks we don’t peer with and because it’s cool.

If you’re delivering 30 Gbps streaming to a network that has limited access, you can deliver just a pinch of it and the rest will be redistributed live P2P between the users.

EuroSport and DailyMotion have been both streaming roughly 1Tbps each like this for a while now.

This is what we did over the course of 3 weeks:

  • Architecture
  • Front-end prototype
  • Monitoring
  • Backend
  • Logic of storages
  • Encoders
  • P2P distribution
  • HTML5/JS player
  • A dozen of cool details: time shift, Apple/Android SDK etc.
Basic technical topology — Green are sources, red are outputs

The funny thing is that Czech Television still uses fuzzy PDC as a format for suppliers to inform about their TV schedule. PDC is based on a more than 30-year-old technology (VPS). Retro. Yet they say a real company always has a technological debt.

Subsequently, after about 2 months of silence, CT decided we won’t get the deal. Even though we offered the lowest price (main criterion) and delivered a superb product. The reason for this? All 3 other competitors have been excluded for formal reasons, that is not responding to additional questions. The contracting authority has a legal right to cancel a tender in case that only one contender remains in the competition. So they cancelled.

We didn’t get the deal, because our competitors didn’t respond to an additional question. Which is understandable since they knew they wouldn’t get the deal.

Now we might not get the deal but we still have the technology and the product. We already made use of the technology with a few of our clients. And we believe there will be more of them so stay tuned for this year.

At first I was sad, CT has a few great tech employees for sure, it’s just that this time bureaucracy won. For a progressive tech company, it’s somewhat a cultural shock dealing with public institutions in general. Anyway, the Czech Republic is still a cool place to be and I’m sure it will be different next time.

As of now, about 4 months later, Czech TV is still working on a new tender and the adjusted rules. However, we started working for several other televisions in the meantime and we can work for you too on live, VOD or hybrid TV delivery via internet and via our global Content Delivery Network.

We now support all modern protocols and technologies ranging from HTML5, HLS, MPEG DASH to MP4, encoding etc. Are you looking for something similar to deliver? Drop us a message: sales@cdn77.com.

Zdenek Cendra, CDN77.com