How to Securely Share AWS S3 Files

This blog post was published on PurpleBox website on Mar 2nd, 2022.

While it becomes attractive for developers, it also becomes an exciting storage place for attackers. Data leaks are becoming extremely critical for companies if the necessary AWS S3 configurations are not made. There are some security points that need to be considered when you’re creating your own AWS S3 bucket:

• Always consider blocking public access first.
• Use AWS S3 encryption.
• Limit your IAM permissions to AWS S3 buckets. Always follow at least the privilege principle.
• Enforce HTTPS to prevent attacks like MITM (Man in the Middle).
• Enhance S3 security using logging.

In this blog post, we will talk about a real-world use case for sharing AWS S3 files securely. We will provide the architecture and configurations for all steps that you will need. Let’s start!

Scenario:

A company wants to share invoice reports weekly. Here are the requirements:

• After a week, reports cannot be reachable for anyone.
• Reports contain critical data like financial details, so they should not be accessible from a public URL.
• The process should be simple as the reporters are not competent in coding.
• The company manager does not want to pay much for the process.

Solution: AWS S3 pre-signed URL generation for every upload!

Let’s review the architecture together:

1.Firstly, we need to create a private AWS S3 bucket for uploading reports. Then we have to create a new API Gateway for uploading reports to the AWS S3 bucket with API requests. So we will be able to upload reports with the API requests.

Our AWS API Gateway should look like below:

We need to get invoke URL from the created AWS API Gateway:

Note: Keep this invoke URL for the next steps. You will put the object in the AWS S3 bucket.

Send the API request to invoke URL for upload reports to AWS S3 (We’re using Postman for this) :

API request body:

Note: This API is accessible for anyone. Attackers love unauthorized access. To prevent this, we need to do allow only specific IP addresses access to our Amazon API Gateway. These IP addresses can be VPN IPs of the company. You can use resource policies for this.

2.We’re checking the reports are uploaded to AWS S3 successfully.

3.We have uploaded reports via API request to create an AWS S3 bucket. Now we need an AWS Lambda function that invokes uploaded reports. This function will generate a pre-signed URL that will expire after 7 days and send this to employees that are defined on AWS SNS. Employees need to be subscribed to the AWS SNS topic before this operation.

Note: All S3 objects are private by default. They can only be accessed by the object owner. By creating a pre-signed URL, the object owner can share objects with others. To create a pre-signed URL, you should use your own credentials and grant time-limited permission to download the objects. The maximum expiration time for a pre-signed URL is one week from the time of creation and there is no way to have a pre-signed URL without expiry time.

You can use the Lambda function code below:

import json
import boto3
def lambda_handler(event, context):
    s3_bucket_name = str(event['Records'][0]['s3']['bucket']['name'])
    s3_report_key = str(event['Records'][0]['s3']['object']['key'])
    s3_client = boto3.client('s3')
    report_presigned_url = s3_client.generate_presigned_url('get_object', Params={'Bucket': s3_bucket_name, 'Key': s3_report_key}, ExpiresIn=604800)
    MY_SNS_TOPIC_ARN = "{SNS_topic_ARN}"
    sns_client = boto3.client('sns')
    sns_client.publish(
        TopicArn=MY_SNS_TOPIC_ARN,
        Subject = 'Reports Presigned URLs',
        Message="Your Reports are ready:\n%s" %report_presigned_url)
    print('success')
    return {'message': 'it works'}

We need to attach the required policies to the AWS Lambda. For test purposes, full access policies are attached. If you’re using this code in production, you need to create your own policies with at least a privilege principle.

4. Now, we should add the event notification that will trigger the Lambda when the report is uploaded to the s3 bucket. We need to navigate “Amazon S3 → {bucket_name} → Properties → Event notifications” and create a new event notification:

We need to select the Lambda function’s ARN that we created before:

5. We should also check reports on the S3 bucket and delete reports whose creation date is more than 7 days. For this, we will create a lambda function and make it trigger every 7 days from AWS CloudWatch. You can use the Lambda function code below:

import json
import boto3
import time
from datetime import datetime, timedelta
def lambda_handler(event, context):
    old_files = []
    s3 = boto3.client('s3')  
    try:
        files = s3.list_objects_v2(Bucket={bucket_name})['Contents']
        old_files = [{'Key': file['Key']} for file in files if (file['LastModified'].strftime('%Y-%m-%d')) < ((datetime.now() - timedelta(days=7)).strftime('%Y-%m-%d'))]     
        if old_files:
            s3.delete_objects(Bucket={bucket_name}, Delete={'Objects': old_files})          
			return {
            'statusCode': 200,
            'body': "Expired Objects Deleted!"
            }       
    except:
        return {
            'statusCode': 400,
            'body': "No Expired Object to Delete!"
            }

When we create the CloudWatch Event Rule, we need to select the event source and targets.

6. All processes are done and ready to use. We can upload reports with the API request. After we send a request to our API successfully, we’ll be able to get a pre-signed URL in an e-mail which will look like this:

When we click on the pre-signed URL, we will be able to download the report we have uploaded with the API request.

Also, the event rule we created in CloudWatch will be checking our S3 bucket weekly and will delete those that have been there for more than 1 week.

In this blog, we’ve summarized what AWS S3 is. We also prepared a real-world use case for sharing AWS S3 files securely. We provided the architecture and configurations for all steps that you need. We hope you enjoyed it! Stay safe in the cloud!

If you want to read more on this topic, feel free to check out the PurpleBox Blog Section.