Schedule automatic Backup in Dynamodb.

Hello People,

Then of Amazon Reinvent:2017 , They created Dynamodb backup this backup solution is very good but they didn’t create scheduled backup.

This article explain as create a lambda function for autobackup and configure custom retention and alerts.

History:

Boss(Luis) says: it is possible scheduler the backup? 
My(Ezequiel): At the moment is not possible.
Boss(Luis): I think is possible create backup job using one lambda function, please review this problem.
My(Ezequiel): Okey perfect! I reading about running awscli commands in lambda but i wasn’t convinced. I keeping investigate and found boto3 support for backups in dynamodb in this url:
http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html
My(Ezequiel): I think this is great but I dont understand as building scripts in python. I’ll try.
MY(Ezequiel): In the last month take two courses in python, I understood a little. 
My(Ezequiel): I began work in the script in four hours this script successfull but didn’t work perfect, later my teammate Juan helped me. He was me teach about functions in python and try/except and Juan review my script.
My(Ezequiel): In twenty minutes more later. It worked perfect. I talked Juan you are a genius, thanks.
Boss(Luis) says: Please add e-mail notification and generated push notification depending on the status.
My(Ezequiel): Okey perfect! In fifty minutes this work was complete.

Requeriments:

  • Aws account and dynamodb table for backup.

Starting to Work:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ses:SendEmail",
"ses:SendRawEmail"
],
"Resource": "*"
}
]
}

Go to create Lambda function and select role created in last step.

In this part , you have two ways:

one: only use e-mail for errors.
two: use email for inform if backup is succesful or push notification if backup is failed.

Default: The script have a one week at retention, it is customizable in this line.

check = dynamo.list_backups(
TableName=name,
Limit=100,
TimeRangeUpperBound = current_time - timedelta(days=7)
#TimeRangeLowerBound = datetime(2015, 1, 1)
)

code for way one(use email alldays if result is succesful and use push notification if result is failed).

import boto3
import sys
from datetime import datetime, timedelta
import calendar
import json
import gc
from pprint import pprint
dynamo = boto3.client('dynamodb')
ses = boto3.client('ses')
sns = boto3.client('sns')
email_from = 'e_ariel@hotmail.es'
email_to = 'earielli@itshell.org'
email_cc = 'e_ariel@hotmail.es'
emaiL_subject = 'Dynamodb backup sucessful in the '
email_body = 'Dynamodb backup sucessful in the '
current_time = datetime.now()
def make_backup(name):
try:
response = dynamo.create_backup(
TableName=name,
BackupName=name+'_bkp_'+ '%s-%s-%s_%s.%s.%s' % (current_time.year, current_time.month, current_time.day,current_time.hour, current_time.minute, current_time.second)
#time.strftime("%Y%m%d%H")
)
print(response)
send_email(name)
return 0
except:
send_sns()
sys.exit("Se produjo un error realizando el paso backup!")

def delete_backup(name):
try:
print("Deleting")
print(current_time)
print(current_time - timedelta(days=7))

check = dynamo.list_backups(
TableName=name,
Limit=100,
TimeRangeUpperBound = current_time - timedelta(days=1)
#TimeRangeLowerBound = datetime(2015, 1, 1)
)

print(check)
for backup in check['BackupSummaries']:
arn = backup['BackupArn']
print("ARN to delete: "+arn)
deletedArn = dynamo.delete_backup(
BackupArn=arn
)
print(deletedArn['BackupDescription']['BackupDetails']['BackupStatus'])
except:
sys.exit("Se produjo un error realizando el paso de limpieza de backups viejos!!")
def send_sns():
sns_message = sns.publish(
TopicArn='arn:aws:sns:us-west-2:881653854182:dynamodb_bkp_failed',
Message='Dynamodb backup failed please check!'
)

def send_email(name):
response = ses.send_email(
Source = email_from,
Destination={
'ToAddresses': [
email_to,
],
'CcAddresses': [
email_cc,
]
},
Message={
'Subject': {
'Data': emaiL_subject + name + ' table'
},
'Body': {
'Text': {
'Data': email_body + name + ' table'
}
}
}
)
# response is a function for dynamodb this develop the backup of table.
def lambda_handler(event, context):
print(event['TableName'])
if make_backup(event['TableName']) == 0:
delete_backup(event['TableName'])

Save function and generated event with your TableName.

Saved and Test function.

Check the backup in Dynamodb. It is working ,please review your e-mail.

check e-mail

In this moment i go to create rule for schedule backup .

In targets select your LambdaFunction and insert the table name.

Now i go configure sns and push alert by opsgenie.

copy your arn in the part sns of the script:

It’s worked!

code for way two(only use email if result is failed):

import boto3
import sys
from datetime import datetime, timedelta
import calendar
import json
import gc
from pprint import pprint
dynamo = boto3.client('dynamodb')
ses = boto3.client('ses')
email_from = 'e_ariel@hotmail.es'
email_to = 'earielli@itshell.org'
email_cc = 'e_ariel@hotmail.es'
emaiL_subject = 'Dynamodb backup Failed! in '
email_body = 'The backup for dynamodb failed! Please review '
current_time = datetime.now()
def make_backup(name):
try:
response = dynamo.create_backup(
TableName=name,
BackupName=name+'_bkp_'+ '%s-%s-%s_%s.%s.%s' % (current_time.year, current_time.month, current_time.day,current_time.hour, current_time.minute, current_time.second)
#time.strftime("%Y%m%d%H")
)
print(response)
return 0
except:
send_email(name)
sys.exit("Se produjo un error realizando el paso backup!")

def delete_backup(name):
try:
print("Deleting")
print(current_time)
print(current_time - timedelta(days=7))

check = dynamo.list_backups(
TableName=name,
Limit=100,
TimeRangeUpperBound = current_time - timedelta(minutes=1)
#TimeRangeLowerBound = datetime(2015, 1, 1)
)

print(check)
for backup in check['BackupSummaries']:
arn = backup['BackupArn']
print("ARN to delete: "+arn)
deletedArn = dynamo.delete_backup(
BackupArn=arn
)
print(deletedArn['BackupDescription']['BackupDetails']['BackupStatus'])
except:
sys.exit("Se produjo un error realizando el paso de limpieza de backups viejos!!")

def send_email(name):
response = ses.send_email(
Source = email_from,
Destination={
'ToAddresses': [
email_to,
],
'CcAddresses': [
email_cc,
]
},
Message={
'Subject': {
'Data': emaiL_subject + name + ' table'
},
'Body': {
'Text': {
'Data': email_body + name + ' table'
}
}
}
)
# response is a function for dynamodb this develop the backup of table.
def lambda_handler(event, context):
print(event['TableName'])
if make_backup(event['TableName']) == 0:
delete_backup(event['TableName'])

# add this line 
This line send a test e-mail, later delete. if your backup failed “lambda” will send e-mail for you.

Thanks for reading this article. Hope you find it helpful 😃