Automating DynamoDB Backups to S3 using AWS EventBridge Scheduler and Terraform

Regular backups of your DynamoDB tables are essential for ensuring data reliability, disaster recovery, and compliance. In this blog post, we’ll explore an efficient and automated way to backup your DynamoDB data to Amazon S3 using AWS EventBridge Scheduler and Terraform.

Manual Backup with AWS CLI

According to AWS documentation you can easily create a one time full backup to S3 with the following CLI command (Thanks Jaroslaw Brodowski for pointing out this feature!) :

aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:us-west-2:123456789012:table/MusicCollection \
  --s3-bucket ddb-export-musiccollection-9012345678 \
  --s3-prefix 2020-Nov \
  --export-format DYNAMODB_JSON \
  --export-time 1604632434 \
  --s3-bucket-owner 9012345678 \
  --s3-sse-algorithm AES256

This command is ideal for creating a one-time backup quickly. Despite its utility, remember that manual processes are error-prone and not scalable for regular intervals.

Automating with AWS Lambda and EventBridge

One common pattern to automate this process is by using an AWS Lambda function to execute the above command and trigger it using AWS EventBridge, as described in this blog.

While AWS Lambda functions triggered by EventBridge are a conventional choice for automating backups, they involve additional complexities such as code deployment and monitoring.

Credits: https://dev.to/ritaly/how-to-export-aws-dynamodb-data-to-s3-for-recurring-tasks-4l47

Simplifying with AWS EventBridge Scheduler

Introduced in late 2022, AWS EventBridge Scheduler offers a more straightforward solution. This tool allows you to call over 270 services and over 6,000 API actions with AWS SDK targets directly. With EventBridge Scheduler, you can eliminate the need for an AWS Lambda function and call the DynamoDB API from a scheduled event.

Schedule directly from EventBridge

In Terraform, implementing this solution is as simple as:

resource "aws_scheduler_schedule" "dynamodb_table_backup_music_collection" {
  name = "backup-music-collection"

  schedule_expression = "rate(30 days)" # or something like "cron(0 0 1 * * *)"

  targets {
    role_arn = "arn:aws:iam::123456789012:role/MusicCollectionBackup"
    arn      = "arn:aws:scheduler:::aws-sdk:dynamodb:exportTableToPointInTime"
    input = <<EOF
            {
              "TableArn": "arn:aws:dynamodb:us-west-2:123456789012:table/MusicCollection",
              "S3Bucket": "ddb-export-musiccollection-9012345678",
              "ExportFormat": "DYNAMODB_JSON",
              "S3SseAlgorithm": "AES256"
            }
            EOF
  }
}

Ensure that your role_arn has the necessary permissions to assume scheduler.amazonaws.com, export the DynamoDB table, and write to your S3 bucket. Also, note that the DynamoDB table must have point-in-time recovery enabled for this method to work.

With just a few lines of Terraform, you can set up regular full exports of your DynamoDB table to be stored in your S3 bucket, making your data readily accessible for various use cases.

Cost Considerations for DynamoDB Backups

Ensuring regular DynamoDB backups is crucial, but it’s important to factor in costs. While automated backups to S3 through AWS EventBridge Scheduler provide long-term storage and comply with many regulatory norms, they may incur higher costs for large or frequent backups. An alternative or complementary cost-effective option is DynamoDB’s Point-In-Time Recovery (PITR), which offers continuous, incremental backups for up to 35 days and can provide significant savings if your restoration needs are within this window. By understanding your data recovery requirements and leveraging the right mix of PITR and S3 backups, you can formulate a backup strategy that balances cost-efficiency with operational resilience.

Conclusion

Automating DynamoDB backups to S3 is now more accessible than ever, thanks to AWS EventBridge Scheduler. This solution simplifies the process and provides a cost-effective way to ensure data reliability and recovery. By following the steps outlined in this post, you can streamline your backup strategy and focus on leveraging your DynamoDB data from S3. Give it a try and share your experiences with us.

Read more about The Stepstone Group, the technologies we use or take an inside look at our organisation & processes. Interested in joining us? Check out our careers page.