Cloud-based file transfer at scale

A Live broadcast at our FlyLive Studio in NYC

Enter Lambdas

The source files that needed to be copied are already in the cloud and the destination is AWS S3 for use with CloudFront. Both are ideal scenarios for servers which are also already in the cloud. My first consideration was simply to use a Docker Swarm cluster to manage a collection of microservices. Such an approach would rely on multiple container replicas to ensure distribution across available machine CPU cores. Then I thought, “Why not try lambdas?” After all, file transfer tasks are what’s known in computer science as an embarrassingly parallel problem — given that each file can be individually transferred.

Introducing FLAnts

Using Lambdas, the process behaves like a swarm of ants carrying objects back to an ant colony. And given that the files in question originate from our Flywheel Live production studio — I decided on the name FLAnts (FlyLive Ants). Cheesy — I know! Fortunately, I’m not in marketing…

Message Queuing FTW

As described above, all four processes have a very important feature in common: each relies extensively on message queuing. The class-scanner queues messages, the class-crawler, and segment-transfer both dequeue and queue messages. And the aggregator dequeues messages and generates a transfer report.

Batching

The segment-transfer service reads transfer jobs from a queue and dispatches them to lambda instances. Also, keep in mind that there are multiple segment-transfer instances (replicas) running.

Error handling

Occasionally, a lambda instance will fail to perform a transfer — shocking I know! When that happens the issue can be attributed to a network socket break or timeout. However, this isn’t an issue, because the entire system relies on messaging queuing so it’s just a matter of re-queuing the failed job.

Visualizing the file transfer

To better see the actual behavior of the overall system I decided to build a web-based visualizer. The thought of rendering each of those virtual ants (I mean lambdas) as they transferred file segments — definitely warranted a side-hack.

Transfer speeds

The speed of transferring files using this approach is absolutely staggering. In one of my earlier tests, the system transferred four terabytes of data in two hours and twenty minutes — that’s roughly 523MB per second!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Carlos Justiniano

Carlos Justiniano

Senior Vice President of Technology @ F45 Training. Former VP of Engineering @ Flywheel Sports. World record holder, author, photographer,