Inside our UHD workflow

Last weekend we ran our first streams of the World Cup in Ultra-High Definition and High Dynamic Range and yesterday announced that we would be continuing this with our coverage of Wimbledon’s main court. Earlier, our colleagues in BBC Research and Development have spoken about some of the deep, technical challenges that have been faced.

Meanwhile, in BBC Design + Engineering Media Services we tackled some problems of our own around how to take these concepts and build a stable production process with the ultimate goal of handing over the management of our UHD streams to the production teams.

We’ve learned a lot over a first week of streams and will be following up with a summary of the issues we’ve encountered during matches and the changes we’ve made to reflect them at the end of the tournament. This post introduces the key components of a live stream and what we ended up building out for UHD.

Update 2018–07–19: You can now read the follow-up post about UHD on the BBC Technology + Creativity blog.

For Media Services, a live stream at the BBC generally consists of the following base components:

• Contribution decoding — where the video signal from the outside broadcast site is received back in a BBC site and decoded.
• Distribution encoding — the signal is then encoded into the final format used for distribution to users, including the full adaptive bit-rate (ABR) set of encodes and structure for chunked-HTTP delivery
• Packaging — this encoded media is then packaged into the correct fragmented MP4 containers with appropriate metadata and manifests that can be served to users over HTTP
• Distribution origins — the BBC’s origin servers that downstream caches use to collect streaming content
• Content Delivery Networks — huge third-party networks and caches that provide the scale of delivery needed to handle the BBC’s streams to ISPs and other edge networks
• Internet Service Providers — the companies our audience pay to receive their internet connection at their home or business, or on their mobile devices
• Client devices — your TV, mobile, or PC

For our existing HD and radio workflows, this is a relatively well defined set of services that we have run for several years now. While there’s plenty we want to improve here, and we always are, it works. For example, two years ago, for the UEFA European Championship, managing ~2.4Tbps was a squeeze at multiple points in the delivery chain, but recently we comfortably managed a record-breaking 2.8Tbps of peak traffic during the Royal Wedding. We broke that on Friday by reaching 3.4Tbps during our first World Cup match, and shortly after broke that one too with just over 5Tbps during England’s opening game against Tunisia on Monday.

For UHD however, we’ve had to review every part of this chain. In some areas this involved choosing vendors, for others it involved building new services. After a lot of work across BBC R&D, Broadcast Engineering, Media Services, TV Platforms and the Online Technology Group (OTG), we’ve managed to build our first attempt. It’s far from perfect, but here’s how we ended up where we are for our first iteration.

Contribution Decoding

Video services locally generally use a set of cabling and signalling standards called SDI. Since UHD is essentially four times the amount of data, Quad-SDI is just four SDI signals, each carrying one quadrant of the stream.

As a raw signal, this would be tens of gigabits per seconds over the internet from Russia, so would be infeasible. Instead, a ‘contribution’ encode is sent to us. This is an IP stream that is quickly encoded, optimised for low latency rather than very low bit-rates; for the World Cup we’ve opted for a 120Mbps signal. This is then sent back to the UK, where our contribution decoders convert this back into a Quad-SDI signal output over 4 SDI cables.

Distribution Encoding

The BBC generally prefers to launch new services using cloud services — whether fully managed or running ourselves on cloud infrastructure.

All new ‘regular’ live services we launch, such as World 2020’s Guraji, Telugu, Punjabi and Marathi bulletins, use cloud-based distribution encoding. Meanwhile, we’re hoping to move as much of our legacy on-premise encoding kit onto a cloud product as they reach the end of their useful life.

Unfortunately, for live UHD encoding, most encoding architectures require all ABR representations to be produced on the same instance, and it is very hard to get enough compute resource on one instance in the cloud to achieve this.

Therefore, for this we had to buy physical appliances — each of which (with four GPUs) can support only one of our UHD ABR sets. Even then, we had to remove some of the ABR representations we hoped to offer.

Distribution Packaging

After distribution encoding, we normally use a separate distribution packager to package the encoded segments into the formats required by end client devices. For regular HD content, the encoder delivers this to the packager using Microsoft’s Smooth Streaming format over an HTTP POST. Unfortunately, our encoder does not support HEVC video in a Smooth output (understandably so, as how to do so was only defined in March), so we were unable to use our normal packaging component.

Our saviours here, as is often the case, were our colleagues in BBC R&D. They for some time have had an internal ‘reference’ packager that is used for internal tests and for some small public trials such as Radio 3 in surround and lossless audio that have been run during the Proms in recent years. This packager operates a ‘pull’ model, preparing chunks for distribution as the encoder writes the encoded output to disk, so removes our Smooth dependency. We’ve worked with the team there to get some more physical, real world servers installed, built and working in a largely production-ready state. The R&D packager then outputs the packaged material into an Amazon Web Services S3 bucket.

Distribution Origin

Our distribution origin services for live serve us well. These are caching/load balancing appliances today. Capacity planning for the initial desire of up to two concurrent UHD streams through them though suggested that we would be reaching the limits of what these devices were capable of, so we’ve looked to replace these too.

For on-demand, we have long used our in-house Radix solution. Our OTG colleagues have now set up a new Radix pool for handling UHD content and everything here is now looking pretty good.

Graph of UHD usage patterns for the opening World Cup weekend

Content Delivery Networks & Internet Service Providers

Currently, the BBC’s top streaming bit-rate is 5Mbps for HD content. The 36Mbps we’re offering for UHD is therefore proving to be quite a challenge in terms of distribution capacity.

It’s for this reason that we announced a strictly limited number of entries into the UHD trial service at the same time — there just isn’t enough distribution capacity in the UK if every compatible device tries to watch a UHD stream at the same time. We’re reviewing this number during and between matches to try and find the right balance between offering the trial to as many users as possible and protecting the stability of internet distribution — both for us and our commercial CDN’s other customers. There are a few unknowns we have that are resulting in us being conservative initially:

• The number of audience members who watch in HD via iPlayer who will now watch in UHD instead
• The number of audience members who watch via traditional broadcast methods (Freeview/Freesat etc) that will now watch in UHD instead — this is the first time we are offering higher quality online compared to broadcast so we can’t predict how many people we convert to IP from broadcast
• The accuracy of our bit-rate modelling from earlier trials — we have an understanding of how many devices played each of the different bit-rates based on the earlier trials and our live UHD test loop; but didn’t know how well the model would hold up until we saw our first England match

This ultimately means we’re monitoring very closely and this limit may move up and down as we learn more about how the service is used.

Client Devices

A substantial testing process has been underway to get as many devices as possible on the trial. Additionally, whilst we’ve offered DVB-DASH streams before, UHD is our first DASH-only service so there was a lot of effort around tightening up support and moving more devices over from other streaming formats. The key things we’ve been looking for are stable decoding of a 36Mbps stream and support for either BT.2020 SDR or BT.2100 HLG HDR.

When set-top boxes are involved, we also have to factor in whether the connected display can present the signal — it would be a waste for us to deliver a 36Mbps UHD signal to a panel that can only display a 1280x720 HD image. Therefore, we have had to add code to detect the capability of the connected display and only offer UHD where the connected display is capable. There currently lacks a standard for detecting these capabilities, which made this quite difficult.

Next steps

There will be lots to learn from this summer’s UHD sport events, and we’ll be going back to look at how to improve the broadcast chain. We’ll also be doing a lot more analysis on how UHD changes when and where the audience consumes content.

Oh, and by the way, Media Services is hiring!