Automating EC2 Volume Snapshot creation in AWS
This Blog has moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.
Hardware failures are very common and specially when we deploy our applications in public cloud environment such as AWS, it is recommended to enable regular and automated backup process to prevent the data loss. In AWS, it is easy to automate the backup process using several built-in services. In this document we are going to show how to setup a fully automated EBS volumes snapshot, using CloudWatch Events and Lambda functions for daily backup.
Prerequisites
- AWS account
- IAM user of that AWS account (It is best practice for everything to be done by the IAM user, not from root account)
- IAM user should be authorised to access services for creating this automation task.
Background
In order to take the backup of the volume attached to the EC2 instance, we need to use EBS volume snapshot APIs. AWS recommends to stop the instance (please refer this document) when we take the backup (snapshot) of root volume which holds the boot partition but we can take the backup of other volumes attached directly without stopping the instance. EC2 and EBS services notify the state change events to CloudWatch and we can trigger Lambda functions when such events occur using CloudWatch rules to automate the snapshot process.
How does this automation work?
We are stopping EC2 instance(s) using CloudWatch schedule rule based on a tag key called snapshot. After the instance is stopped, we are creating root volume snapshot of the instance and when root volume snapshot process is completed then we start snapshot process for other attached volume(s) and also we start the instance as we don’t have to wait for other volume(s) snapshot process to be completed.
Create IAM role
Here we are creating IAM role for an AWS service called Lambda. By this role Lambda can access other AWS resources.
Create an IAM role and add this policy into that. To know how to create IAM role and attach policy for a service please refer this blog.
Create Lambda function
Create an AWS Lambda function and place this code into that. While creating lambda function runtime should be Node.js 8.10 and choose previously created role in existing role. To know how to create AWS lambda function refer this blog.
Create CloudWatch event rules
In this automation we will create four Cloudwatch Event rules.
1st rule
Every day at a certain time it will trigger a lambda function to stop EC2 instances.
Steps to create this rule:
- Go to services → CloudWatch → Rules → click on
Create rule
. - Event Source → choose
Schedule
→ set a cron expression(Eg.`30 20 * * ? *
). For getting help to set cron expression refer this document . → Targets → SelectLambda function
→ select previously created lambda function → Configure input → chooseConstant (JSON text)
and put this{“action”:”stopEc2Instances”}
on text field → click onConfigure details
. - Give
Name
which is required,Description
is optional,State
should be enabled → click onCreate rule
.
2nd rule
If an EC2 instance is stopped, it will trigger a lambda function to create attached root volume snapshot.
Steps to create this rule:
- Go to services → CloudWatch → Rules → click on
Create rule
. - Event Source → choose
Event Pattern
→ selectEC2
inService Name
,EC2 Instance State-change Notification
inEvent Type
,stopped
inSpecific state(s)
, chooseAny instance
→ Targets → selectLambda function
→ select previously created lambda function → click onConfigure details
. - Give
Name
,Description
,State
should be enabled → click onCreate rule
.
3rd rule
If an EC2 snapshot is created successfully, it will trigger a lambda function to start the instance and create attached other volume(s) snapshot.
Steps to create this rule:
- Go to services → CloudWatch → Rules → click on
Create rule
. - Event Source → choose
Event Pattern
→ selectEC2
inService Name
,EBS Snapshot Notification
inEvent Type
,createSnapshot
inSpecific state(s)
,succeeded
inSpecific result(s)
chooseAny source
,Any snapshot ID
→Targets → selectLambda function
→ select previously created lambda function → click onConfigure details
. - Give
Name
,Description
,State
should be enabled → click onCreate rule
.
4th rule
If an EC2 snapshot creation is failed, it will trigger a lambda function to send email to notify about the failure task and start instance.
Steps to create this rule:
- Go to services → CloudWatch → Rules → click on
Create rule
. - Event Source → choose
Event Pattern
→ selectEC2
inService Name
,EBS Snapshot Notification
inEvent Type
,createSnapshot
inSpecific state(s)
,failed
inSpecific result(s)
chooseAny source
,Any snapshot ID
→Targets → selectLambda function
→ select previously created lambda function → click onConfigure details
. - Give
Name
,Description
,State
should be enabled → click onCreate rule
.
Create Amazon SNS Topic
In this automation, we are using Amazon SNS service for sending email notification if snapshot creation is failed.
Create an Amazon SNS topic and subscribe to that topic by giving an email in endpoint and then confirm your subscription.
Note: You have to add created Topic ARN in lambda function code.
Update: AWS has release a new service called Data Lifecycle Manager which makes it easy to take the volume snapshots.
Conclusion
Now that we have learnt how to create EC2 volume snapshot automatically. How do we restore volume from that snapshot? If you know the solution then post it as a comment on this blog.
And stay tuned, for my next blogs..