AWS RDS: Disaster recovery

This Blog has moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

RDS is relational database service by Amazon Web Services (AWS), it is a managed service so we don’t have to worry about the underlying Operating System and Database software installation. RDS supports wide variety of databases both open source and commercial with various licensing models. To know more about the service, please visit official AWS documentation here.

In this blog, I will be focusing on Disaster recovery (DR) options for RDS. DR is very important for any database because if our database encountered a unrecoverable failure then there will be data loss and usually it is very hard to backfill/recover full data. Hence preparing a solid DR strategy even before going live is recommended as failures/accidents can happen any time.

DR options available for RDS

DB Instance snapshots

RDS DB snapshots are actually underlying storage volume snapshot not just DB. We can create both automated and manual DB snapshots for RDS. Current supported frequency for default automated snapshots is daily but we can use CLI or API to create manually snapshots at our desired frequency. We can also use a cron jobs or Cloudwatch event rules to schedule creation of snapshots. There might be performance impact during this time so we need to keep this in the mind and schedule the process accordingly. While this can help to recover from failures but this is not real time so we might have few hours of data loss. Also we need to manually restore the database from snapshot so down time is expected.

Can recover data in the following scenarios:

• Data is deleted by bug in the code.
• Hardware failure occurred.
• Mistakenly someone deleted the database instance.

Can’t recover data in the following scenarios:

• AWS Region wide failure unless we copy snapshots into a different region.
• AWS account is compromised.

How to configure automated DB snapshots:

RDS by default enables automated snapshots on daily basis, we can configure number days RDS should keep these snapshots. Also we can set time at which automatic snapshot should be taken. We can disable this option if we don’t want automatic snapshots but it is highly recommended to enable it.

aws rds modify-db-instance --db-instance-identifier mydbinstance    --backup-retention-period numberofday --preferred-backup-window "hh24:mi-hh24:mi"
Use backup-retention-period = 0 to disable automated snapshots.

How to take DB snapshot:

aws rds create-db-snapshot --db-instance-identifier mydbinstance    --db-snapshot-identifier mydbsnapshot

How to copy DB snapshot across region:

aws rds copy-db-snapshot --source-db-snapshot-identifier arn:aws:rds:ap-south-1:123456789012:snapshot:mysql-instance1-snapshot-20180525 --target-db-snapshot-identifier mydbsnapshotcopy  --region ap-southeast-1

How to restore DB from snapshot:

aws rds restore-db-instance-from-db-snapshot --db-instance-identifier mynewdbinstance --db-snapshot-identifier mydbsnapshot

Please note: this will create a new RDS instance so we may have to change DB endpoint in application.

Also note that parameter groups and security groups may have to be changed based on your configuration.

aws rds modify-db-instance --db-instance-identifier mynewdbinstance --db-parameter-group-name mydbparametergroup --db-security-groups "mydbsecuritygroup1" "mydbsecuritygroup2"

Read replicas

RDS Read replica is another DB instance and to which RDS continuously replicates the data from master DB instance in the order of seconds delay. The read replicas which we create on RDS are configured for read access by default and they will automatically switches to read and write access when master DB instance is down or not available due some maintenance work. Cross region read replicas are the DB instance created in a different region. We can also use them for read queries in our application and it will decrease the load on the master DB instance and write queries gets executed faster.

RDS read replicas supports automatic DR with a few application changes. Application should have capability to switch to read replica’s endpoint when master’s endpoint is not reachable.

Can recover data in the following scenarios:

• Hardware failure occurred on the master instance.
• Mistakenly someone deleted the database instance.
• AWS Region failure.
• Only supported on database engines: MariaDB, MySQL, and PostgreSQL

Can’t recover data in the following scenarios:

• Data is deleted by bug in the code as it gets deleted in the read replica also, so we recommend you to keep regular DB snapshots also.
• AWS account is compromised as they can also delete replicas.

How to configure cross region read replicas:

Easiest way to create read replica is AWS RDS console, just select the instance for which read replica to be created and select “Create Read Replica” under “Instance actions” dropdown. We can also do the same using AWS CLI.

aws rds create-db-instance-read-replica --db-instance-identifier ReplicaDBInstanceIdentifier --region ap-southeast-1 --source-db-instance-identifier arn:aws:rds:ap-south-1:123456789012:db:my-mysql-instance

How to promote read replica to full DB instance:

Go to AWS RDS console and select the instance you want to promote and from “Instance Actions” dropdown select “Promote Read Replica”. Here is the way to do using AWS CLI.

aws rds promote-read-replica --db-instance-identifier myreadreplica

Cross account replicas

Right now this option is not directly support by RDS but we can do using AWS Database Migration Service (DMS) for replication from one database instance in one AWS account and to other instance in another AWS account. This service mainly designed for data migration between different database engines but this also it supports continuous replication of data between 2 databases instances.

Can recover data in the following scenarios:

• Hardware failure occurred on the master instance.
• Mistakenly someone deleted the database instance.
• AWS Region failure.
• Only supported on database engines: MariaDB, MySQL, and PostgreSQL.
• Master AWS account is compromised.

Can’t recover data in the following scenarios:

• Data is deleted by bug in the code as it gets deleted in the read replica also, so we recommend you to keep regular DB snapshots also.

How to setup cross account and cross region replicas:
To do so we need to create replication instance in DMS and configure source and target RDS endpoints and create a replication task to continuous copy the data from source to target. This deserves a blog of its own so I have discussed this in depth in my other blog.

Conclusion

We have discussed various disaster recovery options available for AWS RDS and their configuration and limitations.