Schedule automatic Backup in Dynamodb.

Image for post
Image for post

Hello People,

Then of Amazon Reinvent:2017 , They created Dynamodb backup this backup solution is very good but they didn’t create scheduled backup.

This article explain as create a lambda function for autobackup and configure custom retention and alerts.

History:

Boss(Luis) says: it is possible scheduler the backup?
My(Ezequiel): At the moment is not possible.
Boss(Luis): I think is possible create backup job using one lambda function, please review this problem.
My(Ezequiel): Okey perfect! I reading about running awscli commands in lambda but i wasn’t convinced. I keeping investigate and found boto3 support for backups in dynamodb in this url:
http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html
My(Ezequiel): I think this is great but I dont understand as building scripts in python. I’ll try.
MY(Ezequiel): In the last month take two courses in python, I understood a little.
My(Ezequiel): I began work in the script in four hours this script successfull but didn’t work perfect, later my teammate Juan helped me. He was me teach about functions in python and try/except and Juan review my script.
My(Ezequiel): In twenty minutes more later. It worked perfect. I talked Juan you are a genius, thanks.
Boss(Luis) says: Please add e-mail notification and generated push notification depending on the status.
My(Ezequiel): Okey perfect! In fifty minutes this work was complete.

Requeriments:

  • Aws account and dynamodb table for backup.

Starting to Work:

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ses:SendEmail",
"ses:SendRawEmail"
],
"Resource": "*"
}
]
}
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Go to create Lambda function and select role created in last step.

Image for post
Image for post

In this part , you have two ways:

one: only use e-mail for errors.
two: use email for inform if backup is succesful or push notification if backup is failed.

Default: The script have a one week at retention, it is customizable in this line.

check = dynamo.list_backups(
TableName=name,
Limit=100,
TimeRangeUpperBound = current_time - timedelta(days=7)
#TimeRangeLowerBound = datetime(2015, 1, 1)
)

code for way one(use email alldays if result is succesful and use push notification if result is failed).

import boto3
import sys
from datetime import datetime, timedelta
import calendar
import json
import gc
from pprint import pprint
dynamo = boto3.client('dynamodb')
ses = boto3.client('ses')
sns = boto3.client('sns')
email_from = 'e_ariel@hotmail.es'
email_to = 'earielli@itshell.org'
email_cc = 'e_ariel@hotmail.es'
emaiL_subject = 'Dynamodb backup sucessful in the '
email_body = 'Dynamodb backup sucessful in the '
current_time = datetime.now()def make_backup(name):
try:
response = dynamo.create_backup(
TableName=name,
BackupName=name+'_bkp_'+ '%s-%s-%s_%s.%s.%s' % (current_time.year, current_time.month, current_time.day,current_time.hour, current_time.minute, current_time.second)
#time.strftime("%Y%m%d%H")
)
print(response)
send_email(name)
return 0
except:
send_sns()
sys.exit("Se produjo un error realizando el paso backup!")

def delete_backup(name):
try:
print("Deleting")
print(current_time)
print(current_time - timedelta(days=7))

check = dynamo.list_backups(
TableName=name,
Limit=100,
TimeRangeUpperBound = current_time - timedelta(days=1)
#TimeRangeLowerBound = datetime(2015, 1, 1)
)

print(check)
for backup in check['BackupSummaries']:
arn = backup['BackupArn']
print("ARN to delete: "+arn)
deletedArn = dynamo.delete_backup(
BackupArn=arn
)
print(deletedArn['BackupDescription']['BackupDetails']['BackupStatus'])
except:
sys.exit("Se produjo un error realizando el paso de limpieza de backups viejos!!")
def send_sns():
sns_message = sns.publish(
TopicArn='arn:aws:sns:us-west-2:881653854182:dynamodb_bkp_failed',
Message='Dynamodb backup failed please check!'
)

def send_email(name):
response = ses.send_email(
Source = email_from,
Destination={
'ToAddresses': [
email_to,
],
'CcAddresses': [
email_cc,
]
},
Message={
'Subject': {
'Data': emaiL_subject + name + ' table'
},
'Body': {
'Text': {
'Data': email_body + name + ' table'
}
}
}
)
# response is a function for dynamodb this develop the backup of table.
def lambda_handler(event, context):
print(event['TableName'])
if make_backup(event['TableName']) == 0:
delete_backup(event['TableName'])
Image for post
Image for post

Save function and generated event with your TableName.

Image for post
Image for post
Image for post
Image for post

Saved and Test function.

Image for post
Image for post

Check the backup in Dynamodb. It is working ,please review your e-mail.

Image for post
Image for post
check e-mail
Image for post
Image for post

In this moment i go to create rule for schedule backup .

Image for post
Image for post
Image for post
Image for post

In targets select your LambdaFunction and insert the table name.

Image for post
Image for post
Image for post
Image for post

Now i go configure sns and push alert by opsgenie.

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

copy your arn in the part sns of the script:

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

It’s worked!

code for way two(only use email if result is failed):

import boto3
import sys
from datetime import datetime, timedelta
import calendar
import json
import gc
from pprint import pprint
dynamo = boto3.client('dynamodb')
ses = boto3.client('ses')
email_from = 'e_ariel@hotmail.es'
email_to = 'earielli@itshell.org'
email_cc = 'e_ariel@hotmail.es'
emaiL_subject = 'Dynamodb backup Failed! in '
email_body = 'The backup for dynamodb failed! Please review '
current_time = datetime.now()def make_backup(name):
try:
response = dynamo.create_backup(
TableName=name,
BackupName=name+'_bkp_'+ '%s-%s-%s_%s.%s.%s' % (current_time.year, current_time.month, current_time.day,current_time.hour, current_time.minute, current_time.second)
#time.strftime("%Y%m%d%H")
)
print(response)
return 0
except:
send_email(name)
sys.exit("Se produjo un error realizando el paso backup!")

def delete_backup(name):
try:
print("Deleting")
print(current_time)
print(current_time - timedelta(days=7))

check = dynamo.list_backups(
TableName=name,
Limit=100,
TimeRangeUpperBound = current_time - timedelta(minutes=1)
#TimeRangeLowerBound = datetime(2015, 1, 1)
)

print(check)
for backup in check['BackupSummaries']:
arn = backup['BackupArn']
print("ARN to delete: "+arn)
deletedArn = dynamo.delete_backup(
BackupArn=arn
)
print(deletedArn['BackupDescription']['BackupDetails']['BackupStatus'])
except:
sys.exit("Se produjo un error realizando el paso de limpieza de backups viejos!!")

def send_email(name):
response = ses.send_email(
Source = email_from,
Destination={
'ToAddresses': [
email_to,
],
'CcAddresses': [
email_cc,
]
},
Message={
'Subject': {
'Data': emaiL_subject + name + ' table'
},
'Body': {
'Text': {
'Data': email_body + name + ' table'
}
}
}
)
# response is a function for dynamodb this develop the backup of table.
def lambda_handler(event, context):
print(event['TableName'])
if make_backup(event['TableName']) == 0:
delete_backup(event['TableName'])

# add this line
This line send a test e-mail, later delete. if your backup failed “lambda” will send e-mail for you.

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Thanks for reading this article. Hope you find it helpful 😃

Written by

DevOps SR | SRE @Miroculus.com San Francisco, CA www.itshellws.org

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store