How to setup a Video-On-Demand service on AWS

The Growing Need For A Simple VOD Solution

At TrackIt, we have experienced a growing number of content creators looking for an easy-to-use and simple VOD solution they can implement to start sharing their content with users.

As of September 2019, there was no offering currently available in the AWS marketplace that serves as a full-package plug-and-play VOD solution. Some VOD workflows made by AWS are available on GitHub, such as aws-media-services-simple-vod-workflow but as you will discover below these workflows are not as complete as the one presented in this post.

1. API that allows the content creator to perform CRUD operations on the content
2. Content tagging in the S3 buckets
3. Cognito integration for user management
4. Content Metadata stored in DynamoDB
5. White label branding capability
6. Easy installation with just a single command required to deploy the full stack
7. M3U8 support through Apple HLS

Recognizing the need, TrackIt has developed and shared a clean and simple VOD workflow that can be quickly deployed on AWS with minimal technical expertise.

Since we have strong experience with open-source software and AWS solutions, we decided to publish this workflow on GitHub.

https://github.com/trackit/aws-workflow-video-on-demand

Challenges For Content Creators

1. Limited Control Over Video Content

When using AWS, content creators typically choose to store their video content in S3 buckets. Unfortunately, this provides very little control over the content related to the video. For example, a content creator can easily add tags but cannot add more (video metadata) information about the video such as location, name of the video creator, etc.

2. Potential Security Risks

In order to allow viewers to access the videos from a website, content creators will often implement one of two scenarios:

1. Make the S3 bucket public: In this case, everyone can access all of the content — including private videos.
2. Add specific policies to the bucket: This approach is prone to mistakes in configuration that expose the content creators to possible security breaches.

As a result, using these approaches for a website to provide viewers with access to video content risks opening backdoor access into their AWS accounts.

3. Issues With Video Quality/Resolution

In the absence of a purpose-built VOD solution, content creators generally only provide their viewers with videos of a single quality — the original quality of the uploaded video. Thus viewers with low bandwidth connections might have trouble buffering the content, and those with high bandwidth don’t have the option of experiencing higher-quality video.

The VOD Workflow

TrackIt’s workflow has been designed to be an easy-to-implement and elegant VOD solution. Our workflow implements only the minimum services required.

An API gateway is used to make requests, S3 buckets are used to store video content, and AWS Elemental MediaConvert transcodes content and manages format resolutions.

AWS Elemental MediaConvert

AWS Elemental MediaConvert is a file-based video transcoding service that allows users to easily create video-on-demand (VOD) content for broadcast and multiscreen delivery at scale. Using MediaConvert allows content creators to provide users with multiple resolutions for each of the videos.

How It Works

MediaConvert is used for video transcoding.

The source video is converted into two different kinds of output: MP4 and HLS (allowing users access to multiple resolutions they can switch between during playback).

In our use case MediaConvert also generates thumbnails by extracting multiple evenly distributed frames in the video to provide a preview capability.

Details of a job in MediaConvert

Amazon CloudFront

Amazon CloudFront is AWS’ content delivery network (CDN). CloudFront provides content creators a simple and straightforward solution for video delivery with in-built caching mechanisms. A content creator simply needs to create a new distribution and select the right S3 buckets as targets, and CloudFront does the rest of the work.

Distribution in CloudFront

How the Workflow Works

• This workflow creates buckets that will be used to store the video content
• An API is used to manage VOD content
• An input bucket is created to store raw video content uploaded by the content creator
• Once a file is uploaded into this bucket, a Lambda function is triggered which creates a MediaConvert job to transcode the video
• The transcoded videos are then stored in an output bucket
• The API Gateway provides a method for users to make requests to retrieve the list of transcoded videos available in the output bucket

Features of the Workflow

1. API that allows the content creator to perform CRUD operations on the content
2. Content tagging in the S3 buckets
3. Cognito integration for user management
4. Content Metadata stored in DynamoDB
5. White label branding capability
6. Easy installation with just a single command required to deploy the full stack
7. M3U8 support through Apple HLS

Below you will find all detailed information how to launch this workflow from scratch. Also feel free to comment or contact us through the github issues if you have any questions.

Instructions

Pre-requirements

AWS CLI

All the steps require you to have AWS CLI installed.

Your credentials need to be set inside ~/.aws/credentials (cf. example below)

$> cat ~/.aws/credentials
[my_aws_account]
aws_access_key_id = $MY_ACCESS_KEY_ID
aws_secret_access_key = $MY_SECRET_ACCESS_KEY
region = $MY_REGION

S3 Bucket

A bucket will initially be needed for the Serverless framework. This bucket will be used to store data during deployment.

You can create an S3 Bucket (in the same region where you want to deploy your API), you can call it vod-workflow-serverless-dev for example.

Later on the Serverless framework will automatically create other buckets during the deployment.

CloudFront Key Pair

A CloudFront key pair is needed for the pre-signed URLs generation. Instructions are available here (make sure you have access to the AWS root account to perform it).

You will need to save the key pair ID and the private key; they will be used in the next section for the configuration.

NodeJS / NPM

Instructions are available here to install NodeJS and NPM.

Serverless

Instructions are available here to install Serverless.

You can install it with npm

$> npm install -g serverless

Go to your AWS Console and create an access key for Serverless by following this tutorial. There are details on how to use your access key with Serverless below the tutorial.

Configuration

A configuration file named config.json is available to customize this workflow. Two modifications are mandatory (see below), once you have made them, you can rename the other values in this configuration file based on your needs.

Deployment

You need to set as “deployment bucket” the name of the bucket created above — so Serverless can push content to this bucket for deployment. You can also change the region in which you want to deploy the workflow.

Distribution

First, you will need to add information about the key pair generated above into the configuration file:

• “keyPairId” must be the generated key pair ID
• “privateKey” is the content of the private key that you have downloaded (Note: Each new line should be replaced by \n, so the private key can be stored as one single line)

Deployment

Next, perform the first deployment so Serverless can set up everything required.

• Install required libraries

$> npm install

• You can now deploy the workflow with the following command:

$> serverless deploy
$> serverless s3deploy # This command will add events notification support to trigger Lambda for transcoding

Once the first deployment is finished, you will need to go to the AWS Console and search for the CloudFront distribution created with this workflow. A base URL will be displayed. This value needs to be stored as “baseURL” in the configuration file.

Cognito for User Management

Users are managed using Amazon Cognito which is a service that provides authentication, authorization, and user management for web and mobile applications.

The two primary user pools in this solution are:

• vod-workflow-user-pool-users: For the Users
• vod-workflow-user-pool-admins: For the Admins

Create a user in Cognito User Pool

Create a json file (named example-auth.json for the example) and put this inside for now:

{
    "UserPoolId": "$userPoolId",
    "ClientId": "$ClientId",
    "AuthFlow": "ADMIN_NO_SRP_AUTH",
    "AuthParameters": {
        "USERNAME": "$userName",
        "PASSWORD": "$userPassword"
    }
}

You have to connect to the AWS Console and go on Cognito -> User Pool -> “vod-workflow-user-pool-admins” (take this pool if you want to do admins request like post or delete but it’s the same way if you want to create a user in the other pool, get and save your Pool Id).

On the left, click on App Clients (get your App Client id and save it)

Click on show details and click on the box Enable username password auth for admin APIs for authentication (ALLOW_ADMIN_USER_PASSWORD_AUTH) if the box isn’t checked.

To create a user you have to do the following command (replace the $ variables, and note that you will need to change the password):

$> aws cognito-idp admin-create-user --user-pool-id $userPoolId --username $userName --temporary-password $userPassword

You now have to modify the example-auth.json into replacing the different variables and do the following command to initiate your user:

$> aws cognito-idp admin-initiate-auth --cli-input-json file://example-auth.json

Copy the value of Session in the output and do the following command to change the password (replace the $ variables):

$> aws cognito-idp admin-respond-to-auth-challenge --user-pool-id $userPoolId --client-id $clientId --challenge-name NEW_PASSWORD_REQUIRED --challenge-responses NEW_PASSWORD=$newPassword,USERNAME=$userName --session $sessionGet a token from your user

You just have to do the following command and the $IdToken in ouput is your token (it’s only available for a few minutes)

$> aws cognito-idp admin-initiate-auth --cli-input-json file://example-auth.json

Using the API

First, you will need to upload a video by generating an upload URL:

$> curl -X POST -H ‘Content-type: application/json’ -H ‘Authorization: $TOKEN’ -d ‘{“name”: “vod_workflow_demo.avi”}’ http://vod-workflow-api/videos
{
    “upload_url”: “$UPLOAD_URL”
}

Then, once this URL is generated, you can upload your file to S3:

$> curl —v --upload-file vod_workflow_demo.avi $UPLOAD_URL

You can then track the transcoding status by requesting details about your video to the API.

$> curl -H ‘Authorization: $TOKEN’ http://vod-workflow-api/videos
[
    {
        “status”: “Transcoding”,
        “job_id”: “1234567890123-vod123”,
        “id”: “f8b49d03354b410bcc20518787bfc64b”,
        “name”: “vod_workflow_demo.avi”,
        “last_update”: 1234567890
    }
]
$> curl -H ‘Authorization: $TOKEN’ http://vod-workflow-api/videos/f8b49d03354b410bcc20518787bfc64b
{
    “status”: “Transcoding”,
    “job_id”: “1234567890123-vod123”,
    “id”: “f8b49d03354b410bcc20518787bfc64b”,
    “name”: “vod_workflow_demo.avi”,
    “last_update”: 1234567890
}

Depending on your video, transcoding might take some time.

Once the transcoding is done, you can get the transcoded output and thumbnails:

$> curl -H ‘Authorization: $TOKEN’ http://vod-workflow-api/videos/f8b49d03354b410bcc20518787bfc64b
{
    “status”: “Ready”,
    “job_id”: “1234567890123-vod123”,
    “id”: “f8b49d03354b410bcc20518787bfc64b”,
    “name”: “vod_workflow_demo.avi”,
    “last_update”: 1234567890,
    “content”: {
        “mp4”: “$CLOUDFRONT_PRESIGNED_URL”,
        “hls”: {
            “master_low.m3u8”: “$CLOUDFRONT_PRESIGNED_URL”,
            “master_high.m3u8”: “$CLOUDFRONT_PRESIGNED_URL”,
            “master_med.m3u8”: “$CLOUDFRONT_PRESIGNED_URL”
        },
        “thumbnails”: [
            “$CLOUDFRONT_PRESIGNED_URL”,
            “$CLOUDFRONT_PRESIGNED_URL”
        ]
    }
}

3 kinds of content will be available:

• MP4: MP4 version of your video
• HLS: M3U8 playlist files with multiple resolutions (High, Medium & Low)
• Thumbnails: Extracted frames from your video

If you want to store more details about the video, you can add data into DynamoDB using the API:

$> curl -X POST -H ‘Content-type: application/json’ -H ‘Authorization: $TOKEN’ -d ‘{“author”: “myself”}’ http://vod-workflow-api/videos/f8b49d03354b410bcc20518787bfc64b
{
    “status”: “Ready”,
    “job_id”: “1234567890123-vod123”,
    “id”: “f8b49d03354b410bcc20518787bfc64b”,
    “name”: “vod_workflow_demo.avi”,
    “last_update”: 1234567890,
    “author”: “myself”,
    “content”: {
        “mp4”: “$CLOUDFRONT_PRESIGNED_URL”,
        “hls”: {
            “master_low.m3u8”: “$CLOUDFRONT_PRESIGNED_URL”,
            “master_high.m3u8”: “$CLOUDFRONT_PRESIGNED_URL”,
            “master_med.m3u8”: “$CLOUDFRONT_PRESIGNED_URL”
        },
        “thumbnails”: [
            “$CLOUDFRONT_PRESIGNED_URL”,
            “$CLOUDFRONT_PRESIGNED_URL”
        ]
    }
}

API Documentation

Detailed documentation about the API is available here.

Conclusion: A Simple VOD Solution Designed For Rapid Deployment

The VOD workflow TrackIt has developed and published is a clean, elegant, and simple solution for any content creator looking to rapidly deploy a VOD solution and requires only a modicum of AWS-related technical expertise. The instructions provided in this whitepaper allow you to quickly and easily assemble your own VOD solution and have it running right away. We have designed this VOD workflow as a turnkey entry-solution that a content creator won’t need to worry about once it’s set up properly.

Please feel free to contact us for any questions, feedback, or suggestions. TrackIt can also offer customization or augmentation of this solution for your particular needs as part of our full-services AWS consultancy and software development practices.

About TrackIt

TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Venice, CA.

TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.

TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build and deploy a customized solution tailored to your exact requirements.

In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.