Media conversion on AWS lambda with Kotlin and ffmpeg

So much media to convert …

So much media, so much media, so much media. And it ALL needs to be converted. The thought itself is expensive when thinking about a few thousand files, but imagine throwing over 300k music files to be converted all at once using Amazon ec2 or any other platform’s virtual machines. You’d either take weeks to do it or spin up really large servers. But then you’d probably ask why not use Amazon’s elastic transcoder? It’s relatively easy to set up, much cheaper and just works well. And I’d tell you that for most businesses the idea of any form of vendor lock-in is terrible. Then again you’ve read the title of the article and it would seem that I am contradicting myself, but as counter-intuitive as it sounds AWS Lambda isn’t as closed as it may seem. I use the serverless framework to make sure that the converter is as portable as it can be. You’re free to use any other framework or tool to ensure this but serverless has great documentation, is open-source and they strive to maintain compatibility with various cloud services.

Why Kotlin?

I won’t really speak about Kotlin itself but rather the JVM on Lambda. Apart from golang, JVM performance on AWS lambda is extremely consistent and while golang is relatively new to Lambda, the JVM has offered the best consistent performance on the platform. Which is perfect for our use case, because when running any bulk operations, consistent performance means it’s easy to predict the time it would take for the task to finish. Any JVM language is fine for this but I chose Kotlin because that’s what’s used here.

Setting up the Project

As already mentioned I used the serverless framework, and if you have used it before you’ll note that it is written as a node.js cli tool. That’s really not ideal for an entire dev stack based on the JVM, but there’s a way around that.

So if you haven’t already, install node.js. I prefer using nvm to do this but as with all things installation, do as you will. Also note that after the first setup this will not be required during CI or for any other collaborators.

After successfully installing node.js, install serverless and create a project from the Kotlin template if you wish to follow along with Kotlin.

# Install serverless
npm i -g serverless
# Use the template provided by serverless
serverless create --template aws-kotlin-nodejs-gradle --path conv

For now project setup is done but we will tinker with gradle to improve the experience of deploying the code to aws.

The other thing to do is create an IAM user that has enough roles for this specific deploy, below is an export of the policy I used — your mileage may vary.

After setting it up the IAM user make sure that you either use the serverless login command, or export the keys to your terminal like so:

export AWS_ACCESS_KEY_ID=********************
export AWS_SECRET_ACCESS_KEY=*************************************

And with that you have what you need to begin.

The Goal

Before I go further this is what should happen at the end of this adventure

• Upload a file to an input s3 bucket
• Lambda is triggered and converts the file with ffmpeg
• Output is dumped to an output s3 bucket
• We smile and stare at the files :})

Writing the Function

From the template, the only file we need in the src folder is handler.kt. It should call our conversion class like so. Feel free to do this how you’d like.

For the download and upload to s3, it is heavily documented and here is amazon’s documentation over here.

The above assumes that you will write your own download, conversion and upload code. But there are a few caveats to note.

1. Only the tmpfs is writable on lambda, therefore you must the /tmp
2. You must bundle statically compiled ffmpeg and ffprobe in your jar’s resources (Explained below in the example converter)
3. The deployed function must use credentials that allow it to read and write the output and input buckets

Example Converter

I used the ffmpeg-cli-wrapper for my conversion task. This is an example for those who would like to do as I did — mostly. In order to get it to work as expected, we first need to do the compilation ourselves on amazon linux instance/docker container. The easiest way to compile these binaries is to first install docker, and pull the amazon linux image.

docker pull amazonlinux:latest

Then run a container with a mounted working directory.

#This will mount the current directory as /workdir
docker run --rm -it -v $(pwd):/workdir -w /workdir amazonlinux /bin/bash

Place the following files in your working directory, they are bash scripts made for this purpose.

After executing them, you should then run the following to place the binaries in your working directory.

cp /usr/bin/ffmpeg /workdir ; cp /usr/bin/ffprobe /workdir

You should then move them to your jar’s resource directory. Should look like this

Once you have a setup like above using the ffmpeg-cli-wrapper, you’d point to these files like below.

The resources will be placed in the /var/task folder, also make sure that they are executable.

Configure Serverless

This is a simple config, just edit the serverless.yml to the following, customise as you wish and also make sure that your output bucket exists and is properly configured for your IAM user to read and write it too.

We have defined a deployment bucket, this bucket will be where we store our source code. It is recommended to define this because serverless will otherwise assign a random bucket for this purpose. The deployment bucket is used by AWS to provision your container from a cold start. A cold start is the first call of a function after deployment or a function call after a long period of inactivity. There are several strategies to warm up containers to reduce the impact of speed on cold start but it highly depends on your use case for the function.

Deploy the Lambda

Once all the moving parts are assembled to your liking, the command to build and deploy the function from within the project folder.

./gradlew deploy

Test It!

Run the following in your command line to tail execution logs.

serverless logs -f converter -t

Then drop a file to see if it’s converted

Bonus Points

Having the task only run if serverless and node.js is installed is not so nice, also committing binaries to source code is not ideal. So I customised my build.gradle file to make things simpler.

For the ffmpeg binaries, I uploaded them to static file server and I run a task that downloads them during the build if required. To make things simpler I used this plugin called gradle-download-task.

For the node.js and serverless dependencies, there’s a convenient gradle plugin that manages node.js, aptly name gradle node plugin. You can find it here.

This is my resulting build.gradle file:

Simple things :{)

I loved the article, but how do I assemble all this?

Worry not, I have got you covered. I created a lab in the form of a repo that explains everything in steps and the readme of each branch explains everything in detail. The master branch contains the finished product and is ready to deploy. Even contains a download and upload function from s3 that I conveniently left out. The idea is that each branch contains instructions and the final result of each milestone. Step 4 cannot be ignored though — it is not optional.

Here are the steps and their respective branches:

• Step 1 (Setting up the template with serverless ): Step-1-Serverless-Template
• Step 2 (Writing the converter): Step-2-Handler-Customisation
• Step 3 (Building ffmpeg): Step-3-ffmpeg-build
• Step 4 (Deploying to AWS): Step-4-AWD-Credential-Setup

Before I forget here is the repo https://github.com/Radioafricagroup/s3-ffmpeg-kotlin-lab.git

Can I run this in production?

Another good question and the answer is yes, but there are several things to note and the first is performance from a cold start — it’s abysmal. We can alleviate this problem for this specific use case by running dummy conversions to keep the function hot, use a serverless plugin specifically for prevent cold starts or even because it’s a batch action we can just “prepare” by pre-emptively loading up a dummy file to be a 100% sure. If the JVM is a core part of your stack, you are already familiar with it’s startup time and how to deal with it.

The second part is error handling, what happens when a task fails. For this we do have our own secret sauce but we do this by dumping the files into s3 with a queue worker, and if the conversion task fails the error handler throws the conversion task into an error queue. We then deal with the by attempting retries, updating the status of the task on our database etc. Do as you will.

Deployment, putting this in a CI is pretty simple if you adhere to the bonus points, all you’ll need is gradle in your CI engine and you are good to deploy. Also by specifying a deployment bucket like we did earlier automatically versions the deployments by timestamp — which greatly increases visibility as to what’s running in production.

Consistency, this is really key. Apart from cold starts the JVM on lambda is very consistent in it’s performance. Checkout this article that does a pretty good benchmark comparing all the possible runtimes on lambda. For a task like ours, converting a lot of media, this makes the batch tasks a little bit more deterministic.

Cost — theoretically this should cost significantly less than Amazon ec2 and once we run this for a long enough time to compare the cost of this to what we previously had, a follow up article should be available.