Scheduled On-Demand Backups for Amazon DynamoDB tables along all your AWS accounts: simple, cheap, and scalable.
AWS guys recently released On-Demand Backup for Amazon DynamoDB. By using On-Demand Backup, you can create on your own backups of any DynamoDB table without consuming read and writes capacity units. But what do we do when we need to schedule and keep just a certain number of them?. The solution is a mix of: a few lines in Ruby ran by the Cron of your favorite Linux distribution and the use of Tags. It can be applied to one or more AWS accounts.
Solution overview
- First, you need to have a Linux instance running (t2.micro works fine), at least when backups need to be run.
There you need to install Ruby, and aws-sdk Gem. Here aws-sdk Api does the magic by calling dynamodb: create_backup, delete_backup and others operations :)
apt-get update && apt-get install ruby rubygems-integration && gem install aws-sdk2. Create following Ruby scripts under /usr/local/aws_tasks directory, for example.
mkdir -p /usr/local/aws_tasks && cd /usr/local/aws_tasksvim run_dynamodb_backups_all_accounts.rb
#!/usr/bin/env ruby
# Script to run On-Demandb backups of DynamoDB tables with tag (backup:true). After that, a report of status is sent by email.require_relative './library/aws_dynamodb_client_for'
require_relative './library/aws_dynamodb_filter_by_tag'
require_relative './library/aws_postfix_send_email'
require 'time'today = Time.now.strftime("%Y-%m-%d")
path_to_report = "/tmp/dynamodb_backups_all_accounts.rb_#{today}"
report = File.new("#{path_to_report}", "w")
creds = YAML.load(File.read('aws-credentials.yaml'))open("#{path_to_report}", 'a') { |report|
creds.each do |account|
tables_to_be_backed_up = []
dynamodb = aws_dynamodb_client_for(account)
tables_to_be_backed_up = aws_dynamodb_filter_by_tag(account, 'backup', 'true')if tables_to_be_backed_up.length > 0
puts ""
puts "Account #{account[0]}"
report << "\n\n"
report << "** #{account[0]} **\n"tables_to_be_backed_up.each do |table_name|
begin
bkp = dynamodb.create_backup({
table_name: "#{table_name}",
backup_name: "#{table_name}_taken_on_#{today}_by_Infra",
})
rescue Aws::DynamoDB::Errors::ServiceError
puts " *#{table_name} FAILED"
report << " *#{table_name} FAILED\n"
else
puts " *#{table_name} OK"
report << " *#{table_name} OK"
end
end
end
end
}aws_postfix_send_email('ops@your_domain.com','ops@your_domain.com','Reports of DynamoDB Backups', "#{path_to_report}")
vim remove_dynamodb_backups_all_accounts.rb
#!/usr/bin/env ruby
# Script to remove older DynamoDB backups than "retention_days" var.require_relative './library/aws_dynamodb_client_for'
require_relative './library/aws_dynamodb_filter_by_tag'
require_relative './library/aws_postfix_send_email'
require 'time'retention_days = Time.now - (60 * 60 * 24 * 30) # 1 month
creds = YAML.load(File.read('aws-credentials.yaml'))
creds.each do |account|
tables_to_be_backed_up = aws_dynamodb_filter_by_tag(account, 'backup', 'true')
dynamodb = aws_dynamodb_client_for(account)puts "Account #{account[0]}" if tables_to_be_backed_up.length > 0
tables_to_be_backed_up.each do |tabla_nombre|
backups = dynamodb.list_backups({table_name: "#{tabla_nombre}",})
if backups.backup_summaries.length > 0
if backups.backup_summaries[0].backup_creation_date_time < retention_days
puts "#{backups.backup_summaries[0].backup_creation_date_time} backups for #{tabla_nombre}"
bkp_to_delete_arn = backups.backup_summaries[0].backup_arn
bkp_to_delete = dynamodb.delete_backup({backup_arn: "#{bkp_to_delete_arn}",})
end
end
end
end
vim monitor_dynamodb_backups_all_accounts.rb
#!/usr/bin/env ruby
# Script that reports by email about those DynamoDB Backups that could not been run.require_relative './library/aws_dynamodb_client_for'
require_relative './library/aws_dynamodb_filter_by_tag'
require_relative './library/aws_postfix_send_email'
require 'time'today = Time.now.strftime("%Y-%m-%d")
creds = YAML.load(File.read('secrets-sg.yaml'))
creds.each do |account|
backed_up_tables = []
tables_to_be_backed_up = aws_dynamodb_filter_by_tag(account, 'backup', 'true')
dynamodb = aws_dynamodb_client_for(account)if tables_to_be_backed_up.length > 0
tables_to_be_backed_up.each do |table_name|
backups = dynamodb.list_backups({table_name: "#{table_name}",})
if backups.backup_summaries.length > 0
backups.backup_summaries.each do |bkp|
backed_up_tables << table_name if bkp.backup_creation_date_time.strftime("%Y-%m-%d") == today
end
end
end
endnot_backed_up_tables = tables_to_be_backed_up - backed_up_tablespath_to_report = "/tmp/report_dynamodb_backups_#{account[0]}_#{today}"
report = File.new("#{path_to_report}", "w")
if not_backed_up_tables.count > 0
report << "** #{account[0]} **\n"
not_backed_up_tables.each do |bkp_failed|
report << " *#{table_name} FAILED\n"
end
aws_postfix_send_email('ops@your_domain.com','ops@your_domain.com','DynamoDB tables not successfully backed up', "#{path_to_report}")
end
end
At the moment you should have something like this:
$ ls -l /usr/local/aws_tasksmonitor_dynamodb_backups_all_accounts.rb
remove_dynamodb_backups_all_accounts.rb
run_dynamodb_backups_all_accounts.rb
Under library directory, you need to have 3 more small scripts that are called by the previous ones in order not to repeat code.
mkdir -p /usr/local/aws-tasks/library && cd /usr/local/aws-tasks/libraryvim aws_dynamodb_client_for
#!/usr/bin/env rubyrequire 'aws-sdk'
require 'yaml'def aws_dynamodb_client_for(account)
Aws::DynamoDB::Client.new(access_key_id: account[1]['access_key_id'],
secret_access_key: account[1]['secret_access_key'],
region: account[1]['region'])
end
vim aws_dynamodb_filter_by_tag.rb
#!/usr/bin/env ruby
# Script that returns an array with tables's name with Tag(tag_key:tag_value)require_relative 'aws_dynamodb_client_for'
require 'aws-sdk'
require 'time'def aws_dynamodb_filter_by_tag (account, tag_key, tag_value)
tables_to_be_backed_up = []
dynamodb = aws_dynamodb_client_for(account)
tablas = dynamodb.list_tables({}).table_names
tablas.each do |table_name|
table_arn = dynamodb.describe_table({table_name: "#{table_name}",}).table.table_arn
table_tag_list = dynamodb.list_tags_of_resource({
resource_arn: "#{table_arn}",
next_token: "NextTokenString",
})
table_tag_list.tags.each do |tag|
tables_to_be_backed_up << table_name if tag.key == "backup" and tag.value == "true"
end
end
return tables_to_be_backed_up
end
vim aws_postfix_send_email.rb
#!/usr/bin/env ruby
# Script to send email through postfixrequire 'aws-sdk'
require 'mail'def aws_postfix_send_email(from, to, subject, body)
mail = Mail.new do
from "#{from}"
to "#{to}"
subject "#{subject}"
body File.read("#{body}")
end
mail.deliver!
end
3. As we mentioned before, this solution is multi-account, that means it can be applied to one or more AWS accounts. In this example, we are using just two. But you can keep adding as many accounts as you need one after the other in aws-credentials.yaml
vim aws-credentials.yaml
account_one:
access_key_id: ACCESS_KEY_FOR_ACCOUNT_ONE
secret_access_key: xxxxxxxxxx
region: us-east-1account_two:
access_key_id: ACCESS_KEY_FOR_ACCOUNT_TWO
secret_access_key: xxxxxxxxxx
region: us-east-1
4. Now It’s time to have all these backup stuff done, right? :) Set up the timetable that best suits your infrastructure.
#------------------#
# Dynamodb backups #
#------------------#0 5 * * * /usr/local/aws_tasks/run_dynamodb_backups_all_accounts.rb
0 6 * * * /usr/local/aws_tasks/remove_dynamodb_backups_all_accounts.rb
0 7 * * * /usr/local/etermax/aws_tasks/monitor_dynamodb_backups_all_accounts.rb
5. At this point you may wonder what tables are going to be backed up, right?
Open the DynamoDB console, choose the table at issue and then choose Tags. Set there key:value “backup:true”:
6. Once run_dynamodb_backups_all_accounts.rb script has run, you should be able to see the backup taken:
Summary
In this post, we implement a solution to schedule your DynamoDB backups along one or many AWS accounts using Ruby code and a very small Linux instance that must be running only during the timetable defined in Cron.
