All of your replication servers are belong to us

At Signiant, we use AWS DynamoDB. A lot. We have well over 100 DynamoDB tables for our main applications and while we have a good backup strategy for them (and DynamoDB is crazy reliable), I’ve always wanted more. At Signiant, we also believe in hiring co-op students for more interesting work than making the coffee or generating memes.

Thus it was with great delight that our DevOps guys were able to work with an incredibly talented co-op (Jon Seed) recently to implement a great DynamoDB cross region replication solution that is entirely serverless and handles large numbers of DynamoDB tables. This took quite some time to plan and implement but we think it’s generically useful and hence we’ve open-sourced it and decided to write a little about it here. We’re now using it in production to replicate around 30 million items per week across 123 tables. Number of replication errors? zero. Number of servers supporting this? Zero.

(Edit, Oct 2017). We’ve now replicated 2 billion (with a B!) items between us-east-1 and us-west-2 across 175 tables. Still no errors. Still incredible performance

We don’t need no stinkin’ servers

I recall sitting in the audience at AWS re:Invent (Amazon’s AWS conference) in 2014 when AWS Lambda was announced and hearing (and participating in!) the collective gasp when Lambda was announced. The idea of being able to run code without having to manage servers was so fantastic and futuristic, it was like being in a Dr. Who episode!

When we started looking at DynamoDB replication, AWS had a reference implementation using a library they had developed. It just didn’t scale to large numbers of tables and it required us to write code that ran on servers. We bounced around some ideas of using an army of docker containers running on the EC2 container service (ECS) but it seemed so clunky for a large number of tables and we’d still have to manage capacity. Finally, we settled on using a solution involving DynamoDB streams, Lambda, API Gateway and an S3 website for the console. This seemed to be the ideal solution with essentially infinite scalability.

What did we end up with and how does it all work?

Replicator Function

The magic all takes place in the main replicator lambda function. This is a single lambda function that takes in DynamoDB streams from tables and writes them to a corresponding table in another region. The nice thing here is DynamoDB streams and lambda handle things like retries automatically so the actual lambda function is reasonably straightforward. We also write some custom metrics from the replicator function to cloudwatch, the most important being “how far behind are we in replicating a table?”.

This was actually one of the more interesting problems to solve in the whole project since there’s no concept on a stream of how much is processed. The solution we came up with is elegantly simple — compare the timestamps on the original dynamoDB stream event and compare it to the timestamp on the Lambda event. The difference tells us how far behind we are in processing the stream and helps us tune the write throughput on the replica tables. So far, except for large bulk operations on the source tables, we’re at 0 or 1 minute behind for all our tables (and even the 1 is questionable since the event timestamps are rounded!)

Other functions

There’s some other lambda functions and an API gateway API around the solution that do a few things:

• Handle when new tables are added to the replication solution
• Look for new tables according to a prefix (ie. PROD*) and automatically enable replication for these tables
• Generate SNS notifications if we fall too far behind

All of these are well discussed in the Wiki under the design section.

The Console

The console is essentially a front end to the API Gateway endpoints for managing the overall replication solution. Here you can add new tables to replicate, add prefixes of tables to replicate, check the status of a table and check how far behind a replication is for a table. Like the Lambda functions, the console is serverless and runs entirely from an S3 bucket.

What’s the basic steps to implement all this?

There’s a few moving parts to the solution but it’s hopefully been packaged in an easy enough way to deploy. This basic order will get you up and running:

1. Clone the replication solution from github (see below) and npm install it (there’s a pre-processor for the cloudformation template)
2. Create a new cloudformation stack from the resulting cloudformation template and save the outputs from the stack (you’ll need them when deploying the console)
3. Clone the console app from github and create a cloudformation stack using the template (this just creates an S3 bucket and assigns the policy to it)
4. Create a config file and place it in the bucket (see project docs)
5. Build the console app (npm) and copy the resulting content to the bucket. The console should be ‘up’ at this point

The next step assumes you have existing tables you need to replicate. Basically, you need to enable the dynamoDB stream on the source table(s) first, copy the data and THEN turn on replication for the table. Note that you only have 24 hours to replicate the initial copy of the data since the DynamoDB stream will only look back 24 hours. In our case, we used a solution using Elastic Map Reduce (EMR) to do the initial table copies (see below for github solution for this) but you can copy the data any way you like (monkeys and typewriters even!)

Finally, once you’ve done the initial copy, you can enable replication from the console. You can either enable a single table or a prefix to match. If you enable a prefix, you must have performed the initial copy for each table in the prefix before adding the prefix in the console. There will be a bit of a burst when you enable replication so ensure you have enough write capacity on the replica table(s) to handle this. After that, the write throughput on the replica should be very close to the write throughput on the source table.

So where do I get this magic?

Everything is on GitHub and there’s really 3 main components (2 if you don’t need to worry about an initial copy of existing data)

The Replication Solution (installed via Cloudformation)

The Replication Console (installed as an S3 website)

The Initial Table Copy Solution (bit hacky, uses EMR)