Automating S3 File Copying with AWS Lambda

In this blog post, we will explore how to set up an AWS Lambda function to automatically copy newly created files from one S3 bucket to another. This can be a useful task for various scenarios, such as data backup, content distribution, or compliance requirements.

Prerequisites

1. Source S3 Bucket: This is the bucket where the files you want to copy are stored.
2. Destination S3 Bucket: This is the bucket where the copied files will be stored.

Prepare the IAM Policy and IAM Role

To grant the necessary permissions for the Lambda function, we need to create an IAM Policy and an IAM Role.

1. IAM Policy: Create a new IAM Policy named Lambda_S3_Copy_policy with the following configuration:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            // Statement 1: Allow GetObject and DeleteObject actions on objects within the GuardDuty folder
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            // Specify the ARN of the Source folder to allow actions on its objects
            "Resource": "arn:aws:s3:::elg-testbucket/GuardDuty/*"
        },
        {
            // Statement 2: Allow ListBucket action on the Source elg-testbucket bucket
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            // Specify the ARN of the Source elg-testbucket bucket to allow listing its contents
            "Resource": "arn:aws:s3:::elg-testbucket"
        },
        {
            // Statement 3: Allow PutObject action on objects within the destination_folder prefix
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            // Specify the ARN of objects within the destination_folder prefix to allow PutObject action
            "Resource": "arn:aws:s3:::elg-testbucket/destination_folder/*"
        }
    ]
}

1. IAM Role: Create a new IAM Role for the Lambda function and attach the Lambda_S3_Copy_policy to it.

Setup the Lambda Function

1. Create the Lambda Function: Go to the AWS Lambda service and create a new Lambda function from scratch, using Python 3.9 and the IAM role created in the previous step.
2. Configure the Trigger: Once the Lambda function is created, configure the trigger for the Lambda function:

• Bucket: [Source bucket name]
• Event types: PUT
• Prefix: [Location of Guraduty findings]

1. Paste the Lambda Function Code: Paste the following Python code into the Lambda function:

Make the below changes in code.
source_bucket_name = “<Bucket_Name>”
destination_bucket_name = “<Bucket_Name>”
destination_folder = (Prefix=’destination/’)
source_bucket.objects.filter(Prefix=’source/’)

import boto3
import json
from datetime import datetime, timedelta, timezone

s3 = boto3.resource('s3')

def lambda_handler(event, context):
    source_bucket_name = 'centralized-logging'
    destination_bucket_name = 'elg-security-monitoring'  # Update this if needed
    destination_folder = 'GuardDuty/'
    
    # Define the time window for considering new files (2 minutes)
    time_window = timedelta(minutes=2)
    current_time = datetime.now(timezone.utc)

    source_bucket = s3.Bucket(source_bucket_name)

    for obj in source_bucket.objects.filter(Prefix='AWSLogs/'):
        # Check if the object is a file (not a folder)
        if not obj.key.endswith('/'):
            # Get the last modified timestamp of the object
            last_modified = obj.last_modified

            # Calculate the time difference between the current time and the last modified timestamp
            time_difference = current_time - last_modified

            # Check if the file is within the specified time window
            if time_difference <= time_window:
                source_key = obj.key
                destination_key = destination_folder + source_key.split('/')[-1]

                try:
                    print(f'Copying file {source_key} to {destination_key}')
                    copy_source = {'Bucket': source_bucket_name, 'Key': source_key}
                    s3.Object(destination_bucket_name, destination_key).copy_from(CopySource=copy_source)
                    print(f'Successfully copied {source_key} to {destination_key}')
                except Exception as e:
                    print(f'Error copying file {source_key} to {destination_key}: {e}')

    return {
        'statusCode': 200,
        'body': json.dumps('Newly created files copied successfully!')
    }

This Lambda function will automatically copy any new files created in the GuardDuty/ prefix of the source S3 bucket to the destination_folder/ prefix of the same bucket within a 2-minute time window.

Note: If your S3 bucket is encrypted so add the “AWSKeyManagementServicePowerUser” policy to the IAM role attached to Lambda and update the KMS Key policy.

{
    "Sid": "Allow decryption for CopyObject operation",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:sts::21112560xxxx:assumed-role/Copy-GuardDuty-Dynamic-content-role-qxxx/Copy-GuardDuty-Dynamic-content"
    },
    "Action": "kms:Decrypt",
    "Resource": "arn:aws:kms:<region>:<account-id>:key/<key-id>"
}

Conclusion

By setting up this AWS Lambda function, you can automate the process of copying newly created files from one S3 bucket to another. This can be particularly useful for tasks like data backup, content distribution, or compliance requirements. The IAM Policy and IAM Role ensure that the Lambda function has the necessary permissions to perform the file copy operation.

Remember to replace the placeholders in the code (e.g., elg-testbucket, destination_folder) with your actual bucket names and folder structures. Additionally, you can customize the time window or other parameters to suit your specific use case.