Automating AWS Snapshot Cleanup with Lambda for Cost Optimization

source: Google

One common oversight in Amazon Web Services (AWS) is the accumulation of EBS snapshots. These snapshots are often created for backup purposes when developers work with EC2 instances. However, when these instances are terminated, the snapshots often get left behind, incurring unnecessary costs. This article explores a solution to this problem by leveraging AWS Lambda to automate the cleanup of orphaned snapshots, thus optimizing costs.

Developers frequently create snapshots of EBS volumes attached to EC2 instances for backup and recovery purposes. However, once the corresponding EC2 instances are terminated, these snapshots can become orphaned. Orphaned snapshots are snapshots that are no longer associated with an active EC2 instance. If not managed properly, these snapshots can accumulate, leading to increased storage costs without providing any value.

The solution involves the creation of a "Smart Lambda" function designed to automatically identify and delete orphaned EBS snapshots. This function periodically scans for snapshots, checks if they are associated with any active EC2 instances, and deletes the snapshots if no associations are found. This automated process helps maintain cost efficiency by ensuring that only necessary snapshots are retained.

Implementation

Prerequisites

• Basic knowledge of AWS services such as EC2, EBS, Lambda, IAM, and CloudFormation.
• AWS CLI installed and configured on your local machine.

Step 1: Setting Up IAM Role for Lambda

Before creating the Lambda function, we need to set up an IAM role that grants the necessary permissions for the function to access EC2 snapshots.

1. Go to the AWS IAM Console.
2. Create a new role and select "AWS service" as the type of trusted entity, then choose Lambda.
3. Attach the following policy to the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeSnapshots",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        }
    ]
}

1. Name the role (e.g., LambdaSnapshotCleanupRole) and create it.

Step 2: Creating the Lambda Function

1. Go to the AWS Lambda Console.
2. Click on "Create function" and select "Author from scratch".
3. Name your function (e.g., SnapshotCleanupFunction).
4. Select Python 3.x as the runtime.
5. Choose the IAM role created earlier (LambdaSnapshotCleanupRole).
6. Click on "Create function".

In the function code, paste the following Python script:

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')
    snapshots = ec2.describe_snapshots(OwnerIds=['self'])['Snapshots']
    
    for snapshot in snapshots:
        try:
            if 'InstanceId' not in snapshot['Description']:
                print(f"Deleting orphaned snapshot: {snapshot['SnapshotId']}")
                ec2.delete_snapshot(SnapshotId=snapshot['SnapshotId'])
        except Exception as e:
            print(e)

This script fetches all snapshots owned by the account, checks if the InstanceId is mentioned in the description (a simple way to determine if it's potentially associated with an instance), and deletes the snapshot if it seems orphaned.

Step 3: Automating with CloudFormation

Create a CloudFormation template to automate the deployment of the Lambda function and the necessary permissions.

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  SnapshotCleanupFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Role: arn:aws:iam::[YOUR_ACCOUNT_ID]:role/LambdaSnapshotCleanupRole
      Runtime: python3.8
      Code:
        ZipFile: |
          import boto3
          def lambda_handler(event, context):
              ec2 = boto3.client('ec2')
              snapshots = ec2.describe_snapshots(OwnerIds=['self'])['Snapshots']
              for snapshot in snapshots:
                  try:
                      if 'InstanceId' not in snapshot['Description']:
                          print(f"Deleting orphaned snapshot: {snapshot['SnapshotId']}")
                          ec2.delete_snapshot(SnapshotId=snapshot['SnapshotId'])
                  except Exception as e:
                      print(e)
      Timeout: 120

Outputs:
  LambdaFunctionARN:
    Description: "The ARN of the Lambda function"
    Value: !GetAtt SnapshotCleanupFunction.Arn

Replace [YOUR_ACCOUNT_ID] with your actual AWS account ID.

Step 4: Scheduling the Lambda Function

To ensure that the cleanup operation runs automatically, you can use Amazon CloudWatch Events to trigger the Lambda function on a schedule (e.g., nightly).

1. Go to the Amazon CloudWatch Console.
2. Create a new rule in the Events section.
3. Set the schedule expression (e.g., cron(0 4 * * ? *) to run at 4 AM UTC).
4. Choose "Lambda function" as the target and select your SnapshotCleanupFunction.
5. Configure details and create the rule.

By leveraging AWS Lambda and CloudFormation, you can automate the cleanup of orphaned EBS snapshots, ensuring that you only pay for the storage you actually need. This solution not only optimizes costs but also contributes to a cleaner and more manageable AWS environment. Regularly monitoring and cleaning up unused resources is a key practice in cloud cost optimization.