Using AWS for Video Transcoding and Fast Delivery Streaming

Petr Kout
Petr Kout
Mar 3, 2018 · 9 min read
Image for post
Image for post

Twitter: @petrkout
Web: http://petrkout.com.

There are several compelling platforms out there for streaming an optimized video content to your website visitors’ devices, be it laptops, tablets or smartphones. AWS, with its S3 buckets, Elastic Transcoder, and CloudFront is certainly one of the top contenders. In this article, we will configure an AWS account to accept video files via an S3 bucket, optimize them for streaming to a smartphone, and deliver them with minimal latency through Amazon’s content delivery network CloudFront. Let’s get started!

The Process — What We’ll Learn

  1. We’ll create two S3 buckets: One for storing the video files we want to make available for streaming, one for storing a version of the videos that is optimized for streaming to smartphones and tablets.
  2. We’ll create an Elastic Transcoder pipeline for optimizing the uploaded videos for streaming.
  3. We’ll create an AWS Lambda function to automatically push each uploaded video file through the Elastic Transcoder pipeline with the goal of optimizing the video for streaming and storing the final video in the second S3 bucket.
  4. We’ll configure CloudFront to distribute the optimized video files from the second S3 bucket through the content delivery network (CDN) for low latency streaming, e.g. minimal lag in delivery.

Once these components are in place, they will trigger one another in the following sequence: A video file is uploaded to the first S3 bucket. That event triggers the Lambda function, which creates a job in the Elastic Transcoder pipeline. Once the transcoding job is finished, the transcoded file is stored in the second S3 bucket. As soon as a user initiates streaming of the transcoded video, CloudFront loads the video from the second S3 bucket and streams it to the user’s device.

Image for post
Image for post

Let’s go through the steps:

Prepare Two S3 Buckets

Go the S3 section of your AWS console and create two buckets, one named “streaming-video-raw”, the other “streaming-video-optimized”.

Image for post
Image for post

As you go through the S3 bucket creation wizard, leave the properties and permissions at their default values.

Now, we should have our two S3 buckets ready.

Image for post
Image for post

Also, in order to be able to play the video from the second S3 bucket, we have to make it public, that is, allow reading the content from the bucket by any Internet user. To do that, go to the bucket and click on the “Permissions” tab. Then click on the “Bucket Policy” button. A screen shown below will open with a text editor to enter the policy configuration.

Image for post
Image for post

Enter this code into the policy editor:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::streaming-video-optimized/*"
}
]
}

Click the “Save” button. Moving on…

Create Elastic Transcoder Pipeline

Navigate to the Elastic Transcoder AWS service and click on the “Create New Pipeline” button.

Image for post
Image for post

The screenshot below shows the configuration parameters for the new pipeline:

Image for post
Image for post

The pipeline is essentially a queue of transcoding jobs; it takes video files from the input bucket one at a time, transcodes them to the desired format, which will be specified later in the Lambda function, and stores the transcoded files in the output bucket specified above. It will also create a thumbnail image for each video and store it in the output bucket along with the video.

Create AWS Lambda Function to Create Transcoder Pipeline Jobs

To create the Lambda function and connect it to the S3 bucket “Create Object” event — the event triggered every time a file is uploaded to the bucket — go to the AWS Lambda service section, click on “Create function” and fill the form like so:

Image for post
Image for post

From the “Policy templates” drop-down box, select “Basic Edge Lambda permissions”. Click on the “Create function” button at the bottom to submit the form. You’ll see a screen like this:

Image for post
Image for post

From the list of triggers on the top left, select S3. Then scroll down to the second half of the page and configure the connection between the Lambda function and the S3 bucket. The below screenshot shows the configuration:

Image for post
Image for post

We are telling Lambda to trigger the function when a new object (file) is created in the “streaming-video-raw” S3 bucket. Click “Add” to save the changes.

Next, click on the “transcodeVideo” function name in the rectangle in the center top of the screen. A function editor will open in the lower half of the screen. Enter the following code there. We will just change a few parameters:

var AWS = require('aws-sdk');
var s3 = new AWS.S3({
apiVersion: '2012–09–25'
});
var eltr = new AWS.ElasticTranscoder({
apiVersion: '2012–09–25',
region: 'us-east-1'
});
exports.handler = function(event, context) {
var bucket = event.Records[0].s3.bucket.name;
var key = event.Records[0].s3.object.key;
var pipelineId = '1519927986763-h63mfd';

if (bucket !== 'streaming-video-raw') {
context.fail('Incorrect input bucket');
return;
}

var srcKey = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, " ")); // the object may have spaces
var newKey = key.split('.')[0];
var params = {
PipelineId: pipelineId,
OutputKeyPrefix: newKey + '/',
Input: {
Key: srcKey,
FrameRate: 'auto',
Resolution: 'auto',
AspectRatio: 'auto',
Interlaced: 'auto',
Container: 'auto'
},
Outputs: [{
Key: newKey + '.mp4',
ThumbnailPattern: 'thumbs-' + newKey + '-{count}',
PresetId: '1351620000001-100020',
Watermarks: [],
}]
};

console.log('Starting a job');
eltr.createJob(params, function(err, data) {
if (err){
console.log(err);
} else {
console.log(data);
}
context.succeed('Job successfully completed');
});
};

When you paste the above code to your Lambda function editor, replace the pipelineId variable with the with ID of the pipeline in your account. You can get the pipeline ID by using the AWS terminal command like so:

aws elastictranscoder list-pipelines

The output JSON string will contain the pipeline Id field. Paste its value into your AWS Lambda function. The other variable you might want to change in the future is the PresetId . The value the above code sample contains refers to a preset that transcodes the video to a progressively streaming MP4 file optimized for smartphones and tablets. If AWS changes the Id of the preset in the future, you might need to get the new value. You can retrieve it by running this command:

aws elastictranscoder list-presets

Scroll through the JSON output and find the Id of the preset called “System preset: iPhone 4s and above, iPad 3G and above, iPad mini, Samsung Galaxy S2/S3/Tab 2”. Set the value of the PresetId variable in our function code and click on “Create function” to submit the form. Your function is now ready.

Also, before saving the AWS Lambda function, scroll down and set the execution role to the one we created earlier as shown on the following screenshot:

Image for post
Image for post

Finally, press the “Save” button to create the function.

Currently, the role “videoTranscoderLambdaRole” that is attached to this function doesn’t include the permission to submit jobs to the Elastic Transcoder. Let’s add that policy so that our function can submit jobs to the Elastic Transcoder.

Navigate to the IAM section. There, click on “Roles”. From the list of available roles, click on “videoTranscoderLambdaRole”. On the role detail page, click on the “Attach policy” button and search for the “ElasticTranscoderJobsSubmitter” role. Once found, select it and attach it to the role as shown on the screenshot below:

Image for post
Image for post

Ok, at this point, we are done with the fundamentals. You can try to upload a video file to the “streaming-video-raw” bucket. Wait a while and you should see the transcoded MP4 file appear in the “streaming-video-optimized” bucket.

The last step is distributing the optimized video files that get created in the “streaming-video-optimized” S3 bucket to a content delivery network for speedy delivery to your users’ smartphones. That’s where AWS CloudFront come to play:

Configure CloudFront Distribution

Image for post
Image for post

Set the first few values as shown in the screenshot below; specifically, select the “streaming-video-optimized” S3 bucket as the “Origin Domain Name”. Leave the rest of the settings at their current values (unless you want to change them knowing what you’re doing). Click on “Create Distribution” at the bottom of the page and wait a while (might be up to 20+ minutes) for the distribution to get created. It will show as “In progress” for a while.

Image for post
Image for post

When it’s created, you’ll be able to replace the domain name of the S3 bucket with the domain name of the CloudFront distribution. In this example, the screenshot shows the domain name to be d2fcvogyvoitxb.cloudfront.net. It will be different on your AWS.

Let’s Test It

Image for post
Image for post

Now let’s play the video file through CloudFront. What is the URL? You can determine it by taking the S3 bucket file URL from the S3 bucket as shown on the screenshot above using the red arrow. In our example case, the url is:

https://s3.amazonaws.com/streaming-video-optimized/IMG_0009/IMG_0009.mp4

To convert it to a CloudFront URL, replace the part

https://s3.amazonaws.com/streaming-video-optimized/

with https://d2fcvogyvoitxb.cloudfront.net

so the final link looks like:

http://d2fcvogyvoitxb.cloudfront.net/IMG_0009/IMG_0009.mp4

And that’s it! We have a pipeline that takes video files from an input S3 bucket, converts them to a streaming optimized files stored in an output S3 bucket and distributes the files through a CDN for low latency streaming experience.

References

For more articles, follow me on Medium, Twitter @petrkout, or visit my website at http://petrkout.com.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store