Moving Live Video Quality Control from the Broadcast Facility to the Living Room
The Remote TOC Project
Live streaming has been an integral part of the video technology at Disney Streaming Services (DSS). DSS has multiple teams whose main functions are to preserve the highest possible video quality, and that the content transportation technologies are working as they should 24/7. In order to do this, DSS built and staffed Transmission Operations Centers (TOCs) in San Francisco and New York City, providing state-of-the-art video monitoring systems that allowed the TOC Staff to view and perform quality control on any of the thousands of IP-based streams that we might be processing at a given time.
As cases of COVID-19 in the United States increased, the decision was made to move the DSS workforce home. While this resulted in a variety of challenges for all of our employees, the challenges for the TOC Operators were unique.
The live video content that DSS uses for its sources tends to be high-bitrate, broadcast-quality streams. These streams are intended to be consumed via satellite or fiber. Some of them run in the neighborhood of 100 Mb/s. If people tried to consume these streams in their native format, the streams would be chewing up a huge amount of bandwidth, both on the DSS VPN and on users’ home internet connections. In order to allow TOC Operators to watch this content remotely, we had to generate proxy streams which would look good enough to monitor video, yet also reduce bandwidth to the point that any home internet connection could consume a few of these streams simultaneously. Aside from bandwidth, it was also important to provide these streams as close to real-time as possible, as our operators needed to see the streams “as they are,” not “as they were 30 seconds ago.”
Luckily, DSS has years of experience compressing high-quality video so that it can be consumed on devices at a low bitrate. DSS’s homegrown video transcoder, xCoder (the same transcoder which makes content for Disney+ and ESPN+, among other properties) had a solid foundation on which to build the Remote TOC monitoring platform. If you’ve ever watched anything on Disney+, ESPN+ or MLB.tv, you’d know that the video that xCoder produces looks amazing, so that provided a huge advantage. Using xCoder for this purpose, however, created a new set of challenges.
This isn’t going to be easy…
For both live and linear uses, xCoder was built to consume “groomed” transport streams. These streams are processed by DSS’s Broadcast Infrastructure environment, which standardizes incoming transport streams and sets the structures for Packet Identifiers (PIDs) and Program Map Tables (PMTs) to follow company standards. The advantage of having this environment is that it simplifies the setup of the xCoder for any given transcode job. Each 188-byte packet of data in a transport stream has a packet identifier (PID) that tells a receiving device what it is (video, audio, metadata, etc.). The PMT defines to a receiver of a transport stream what each of these PIDs are. Each elementary stream has its own PID number. With the thousands of streams out there, made by thousands of different encoders, the PID structures of each stream can be unique. Standardizing the structure in our broadcast environment allows the xCoders to just “worry” about decoding and encoding. TOC Operators needed to be able to monitor both groomed and ungroomed streams to make sure that this infrastructure is functioning correctly.
xCoder uses presets to determine the types of content coming in, and also the types of content to encode. One of the goals of the stream standardization system is to utilize as few presets as possible for a given partner. For example, for ESPN+, the system allows us to only have one preset for all events, from UFC to Ivy League Water Polo.
xCoder was built as part of a signal flow that takes in the groomed broadcast contribution sources on one end and produces video and audio segments for HTTP-based streaming on the other. In order to reduce latency, a few steps needed to be removed from that signal flow: the broadcast infrastructure that grooms the streams, the segmenter that generates the segments, the origin, and the CDN. As a bit of background, from an end user’s perspective, the segmenter actually produces and encrypts the media files that the viewer gets (short “chunks” of the video and audio). The CDN acts as a distributor of the media files so that everyone in the world trying to access the same media isn’t requesting from a single point, which would certainly cause a bottleneck. xCoder, as a standalone, produces a proprietary TCP stream. In order for the Remote TOC project to succeed, we needed a way for these streams to be transported to, and decoded by, users on their computers at home.
Stream analysis to the rescue!
xCoder exposes an API hook that allows us to quickly determine things like frame rate, resolution, and even whether or not the xCoder is able to subscribe to the stream. This was originally made to be a troubleshooting tool, but we realized it could be repurposed to be the basis of the Remote TOC’s preset-on-the-fly routine.
Let’s say we have an ungroomed transport stream we wish to monitor with the following PID configuration:
PMT: 500
VIDEO (h.264): 2120
AUDIO (MPEG-1 Layer II): 2220
AUDIO (AC-3 5.1): 2221
SCTE-35: 50This is all information that xCoder’s stream troubleshooting API exposes upon request. After passing the PID information back to the Remote TOC user, we allow them to pick the audio stream (either the MPEG-1 or AC-3) they wish to attach to their Remote TOC proxy transcode, as opposed to just picking the first one present in the PMT. In addition, this feature allows users to see if there are any surprises present, such as unexpected PIDs that the provider may be unintentionally passing to their customers.
…but we have so many streams to monitor
DSS has a complex, highly scaled encoder and resource provisioning system. Its job is to set up all the resources needed to make a live stream and get it to the customer. DSS has thousands of resources dedicated to live-video encoding. Without automation, managing these resources would be impossible. In the early days of the DSS work-from-home effort, we scrambled to build a system to provision streams in a novel way from what was done before in order to utilize the aforementioned stream detection algorithm. The decision was made to have the Remote TOC xCoder provisioning system live on the users’ computers.
DSS was able to dedicate several servers to this new workflow, giving the team’s users the ability to monitor hundreds of streams concurrently. A decision was then made to do simple load balancing in order to keep the streams spread out across the hardware resources:
def bestxCoderToStart():
xcoderList = getxCoderList()
activeJobsDict = {}
#create a dictionary of xcoders in the pool with values containing the number of active encode jobs on the hardware resource
for thisxCoder in xcoderList
activeJobsDict[thisxCoder] = getNumActiveJobs(thisxCoder)
temp = min(activeJobsDict.values())
#This line returns the key with the lowest value in the dictionary
res = [key for key in activeJobsDict if activeJobsDict[key] == temp]
return(res[0])Through this simple function, the resources were balanced, thus ensuring none of them were taxed unless it was absolutely necessary.
It’s great to make the streams, but we still need to be able to watch them
As discussed earlier, an approach was required for stream viewing. Luckily, the framework was already in place for a viewer mostly built for a different use. Repurposing this viewer was not entirely straightforward, as the streams being generated by the Remote TOC xCoders were slightly different; so a fair amount of troubleshooting and testing was required. Eventually, however, the player was compiled and packaged for the users.
Before embarking on this project, DSS’s operators were staffing the TOC 24x7, ensuring that the live content for Hulu Live and ESPN+ looked great and were working as expected. This effort helped them to continue their 24x7 functions from the safety of their own homes, ensuring that the content for Hulu Live, ESPN+, and other properties could continue during this unprecedented global event.
Thanks to Joe Hart and Emanuel Parisi for their efforts making the player work for users at home.
