Cloning databases for production-like environment | RDS Aurora | Cross — Account

Let’s agree to the fact that we need production-like data in one of the test environments, in order to be confident about the feature we develop, reproduce that tricky bug, reliable data-intensive testing, and so on. Now, we can argue that this is not always true but for us, it makes complete sense. We have an application that aggregates data from different sources and makes it available to the end-users for further analysis and action. We solved this problem sometime back and the solution worked like a charm.

This blog is in continuation with the previous blog. We have seen different ways of syncing data between different databases present in the same account (clone, restore from snapshot, and mysqldump). The solution worked well until a fine day we took a call to separate the AWS accounts for dev and prod use cases. Issues faced with separate accounts were:

• Database cloning doesn’t work seamlessly across accounts
• Snapshot restoring isn’t as fast as cloning

Let’s get into the details of how we solve this!

For simplicity let's call the new AWS account as Dev account.

Clone database across accounts

AWS doesn’t allow cloning of databases across accounts directly but we can achieve this indirectly. The question is how can we? Hang on, this is what we are talking about. Simple steps, follow along.

• Share your database cluster using RAM(Resource Access Manager)with the new account

• The cluster is now available in Dev account under shared resources

• Create a clone from the cluster

Clone databases across multiple accounts

Oh, this is simple! You can create multiple clones from a database cluster.

You can’t create more than 15 cross-account clones from any single Aurora DB cluster. Each of these 15 clones must be owned by a different AWS account. That is, you can only create one cross-account clone of a cluster within any AWS account.

Clone multiple databases in single accounts

If this is the case, then you have landed yourself in trouble. AWS doesn’t allow you to create more than one clone in a single account. How to solve this?

• Figure out your most used environment that needs faster sync of data.
• Create clone using the shared cluster from prod account

What happens to other databases in the same account? Improvise!

1. Create a snapshot of the production database.
2. Share snapshot with Dev account
3. Restore the snapshot in the Dev account

Now you must be thinking what’s new with this approach? This is the most common strategy. Well, that’s correct but this has to be automated as we hate anything that needs human intervention. And once automation comes into the picture, some things change. What is the complexity here?

• You can’t share an automated system-generated snapshot. You read that right!
• You need to write a job to create and share the snapshot(manual) every day. Preferably with the date change.

• Write another job that uses the snapshot and restores the database in a new AWS account.

The solutions fit the use cases of our organization. You might go ahead with the snapshot approach or cloning approach depending on the AWS account structure that your organization follows. Let it sync!