S3 to Frame.io Copy Using Lambda

I wanted a solid solution for copying an entire S3 bucket into a Frame.io project that would keep the folder structure and support any file size. Frame.io provides a great Zapier integration for the same task, but it's limited in what file size it can handle and does not recreate folders.

The workflow is built using the Serverless Framework. Therefore, even people without much knowledge of the services involved should be able to get it running with a few commands.

Goals

• Quick copy from S3 to Frame.io
• Handle any file and bucket size
• Preserve folder structure

Architecture

• API Gateway triggers the workflow from a POST request
• Main Lambda verifies project info
• Copy Lambda copies S3 -> Frame.io
• Copy restarts itself if Lambda timeouts

The copy is done by generating a presigned S3 URL to your asset and sending it Frame.io for ingest. Doing it this way greatly reduces Lambda processing time since it only does simple HTTP requests and no handling of the actual bytes in the file.

Setting it up

• Make sure you have npm and docker installed and running
• Clone the github repo and install the Serverless framework

git clone https://github.com/strombergdev/s3-to-frameio-lambda-copy.git
cd s3-to-frameio-lambda-copy 
npm install -g serverless
sls plugin install -n serverless-python-requirements

• Setup AWS Credentials. Checkout this video.

Configure

• Open handler.py and add your Frame.io developer token.

FRAMEIO_TOKEN = 'your_token'

• Deploy it to AWS

sls deploy

After a hopefully successful deployment you will see a url created for your service.

Running it

You can use curl or any other POST request tool to start it. I found Reqbin to be simple to use.

• Paste your URL and make sure it's set to POST
• Click content and enter

{
  "bucket": "your_bucket",
  "project": "your_frameio_project",
  "token": "your_frameio_dev_token"
}

• Click send!

Your copy will start and assets should start to appear on Frame.io! 🎉

Keep in mind that you will be billed for Lambda usage and egress traffic from S3.

There is no way to kill a running lambda, but if you need to stop any further invocations you can use the Throttle button in the Lambda function control panel. To re-enable, run sls deploy again.

To remove the resources from AWS run:

sls remove

The code

See the github repo for full code. Below are some highlights.

The main lambda starts with validating that the bucket and project exist. Then it triggers the copy function using the code below. Note that the trigger is asynchronous (‘Event’), meaning that it will return immediately with the “Copy started” info and the copy function will continue to run on its own.

The special “parent_ids” dict we are creating and passing in is used to know where to upload files. All assets on Frame.io have a parent with a unique ID. We need the ID when we upload a file to know in what project and folder it should go. The dict starts with just the root path and the root_asset_id of the project. Then for every folder we create, we add the folder path and the folders ID to the dict to know where its children should be uploaded.

The other parameters are used if the copy takes longer than the Lambda timeout so a new function has to be started. Then we need to know which asset to continue with.

To fetch all objects from S3, we are using the high-level collections/resources API that does pagination automatically.

When we find a new file, we first call the S3 API to generate a presigned URL, then the Frame.io python API to upload it.

The Serverless definition YAML is used to create all the resources on AWS. Memory size defines how quickly your Lambdas will run and can be tweaked to find a good price/performance tradeoff. The python plugin and docker is used to package all dependencies needed by python and make sure they work on AWS no matter what platform you are developing on.

Only the main function has an event attached to it for triggering since the copy function is triggered directly using its name.

That’s it! Let me know if you have any questions.

/Max