How P7S1 handles serverless content distribution

by Daniele Frasca

Content distribution is one of many use cases for the media and entertainment business, which increases the reach of digital content. This practice is known as “syndication”, where content is distributed to partners that integrate it into their consumer products.

Conceptually speaking, there are three primary operations that a syndication application must handle:

• Ingest
• Update
• Takedown

Each operation typically generates a package made of:

• Video(s)
• Metadata format(s) (XML, XLS, CSV)
• Image(s)

Once the package is generated, it is time to deliver it. The way of content delivery varies, but the most used services are:

• Aspera
• FTPS
• S3

Our starting point architecture was based on a cluster where we had services up and running 24h/7 to generate metadata and image transformation. The internal orchestrator needed to coordinate the package’s generation and, of course, another cluster for the database. This architecture had many flaws starting from scaling issues, reliability, traceability, and cost. It was costly because it kept servers running all the time, even when there was no workload, especially for the primary services where operations could be completed in a matter of milliseconds.

Luckily, we can now leverage the power of serverless architecture, and hence gain benefits like:

• Scalability
• Pay-as-you-go
• Event-driven-architecture

This article will describe each step from when the delivery request is received to the final customer destination.

I will avoid going too deep into each specific step, and I will keep the explanation at a high level. At the end of the article, we can understand how ProSiebenSat.1 distributes content to other media companies and social media platforms.

We could split the functionalities into three main parts:

1. Determine the package content/structure.
2. Generate package.
3. Deliver package.

Those three operations are self-explanatory, but they often need customizations to deliver content according to partner specifications.

Determine the package content/structure

The first step is to understand what kind of package should be delivered and to do so. The logic is as follows:

At this point, when a delivery request arrives, there is some business logic to run to understand what type of package should be generated. This part consists of one Lambda function that will decide if the entry event should be processed from one of the three Lambda Functions:

CREATE

• Never delivered before.
• INGEST package is needed.

UPDATE

• They were already delivered. Metadata, image or video has changed.
• Based on specific logic, INGEST, UPDATE or TAKEDOWN package is needed.

AGGREGATION

• Collect all type of updates to be delivered later based on custom scheduling.

Each of these Lambda functions has one responsibility, and the business logic is driven by specific business rules loaded by AWS AppConfig.

Something to be aware of is this: The content ready event in input has a one-to-many relationship because a video will be distributed to many different platforms. You can receive from one to more than a thousand content-ready events at any given time.

Thanks to the serverless-first approach, we can now, compared to the past, handle pick requests without being worried that the infrastructure will lose a single event in the process of scaling.

Generate package

Once the package is generated, we emit an event with AWS EventBridge that will start executing an AWS StepFunction where dedicated steps will build up the package delivered to the distribution partner.

This AWS StepFunction is a simplified version of the flow we currently have, but it still explains each state’s steps.

TranscodingPreparation:

• Load platform-specific configuration.

OnDemandTranscoding:

• SQS Callback pattern.
• We pass a message that includes a task token to the AWS SQS queue.
• An external service will transcode the video based on the distribution partner’s video profile.
• When the success task token is received, the workflow continues.

TranscodingResult:

• Process and persist the JSON output of the task.

ImageTransformation:

• Generate custom images based on the distribution partner’s image profile.

MetadaGeneration:

• Generate custom metadata file according to partner specifications.

ReadyToBeDeliver:

• Emit event “package ready to be delivered”.

NotifyErrorState:

• We default each state here to handle errors and get alarms.

Something to consider is the quotas for startExecution on the AWS StepFunction. They vary a lot between regions, and the maximum you can get will still not be enough to handle high pick time.

Deliver package

When the previous step function is successfully executed, and the ReadyForDelivery event is emitted, we will start an AWS StepFunction dedicated to the package delivery.

This AWS StepFunction is a simplified version of the flow we currently have, but it explains each state’s steps.

InitiateDelivery:

• We are loading platform-specific configuration.

Delivery:

• We use the AWS Batch integration pattern.
• We download videos, metadata and images from S3.
• Based on partner requirements, we deliver the package through FTPS, IBM Aspera, AWS S3, etc.
• A package transfer can require hours to be completed, especially if the package size goes over hundreds of Gigabytes.

TransferCompleted:

• We emit event Delivered.

RetryInCaseOfError:

• We take care of AWS Batch submitJob quotas and batch failures.

Because we have long-running processes, the AWS StepFunction integration with AWS Batch comes in handy because it takes care of routing jobs on the available batch queues. You need to define a Docker image for the delivery where an AWS EFS or AWS FSx is mounted as a storage service to host temporary data (images, metadata, video). Once configured on the job definition, you are done because there is no software or services to install. The managed part is all done for you, avoiding the complexities of provisioning, managing, monitoring, and scaling your batch computing jobs

Other benefits of AWS Batch are:

• It enables the prioritization of jobs.
• Allow up to 10 retries if the job fails.
• You can use spot or on-demand instances.

Conclusion

In this article, I tried to show you how to leverage cloud-native managed services. You can achieve a cost-efficient, high scalable, secure solution without having deep knowledge in Ops operations. Using AWS cloud-native managed services will allow you to focus on business logic and solve and deliver values faster and more straightforwardly.