Secure File Uploads and Downloads in S3 Using Presigned URLs

When deciding the visibility of an S3 bucket, it’s crucial to consider the security requirements of your application and the intended use of the files. Sometimes, we face conflicting needs: maintaining a private bucket for security while allowing users or applications to read or upload files directly in S3.

Possible solutions include providing AWS credentials or IAM permissions to each entity that needs access, which is impractical, or making the bucket public, which compromises security. A more effective solution is to use presigned URLs. They allow you to keep your bucket private while granting temporary read or upload permissions for specific objects to users or applications. This ensures secure, controlled access without compromising the bucket’s privacy.

In this article, I will focus on using presigned URLs for uploading and downloading files, which are the most common use cases. However, presigned URLs can also be used for other operations. I will explain what an S3 presigned URL is, discuss the GET and PUT presigned URLs, explore various ways to generate them, and conclude with a simple React app example that uploads files to a private S3 bucket using a presigned URL generated by a Lambda function.

S3

S3 (Simple Storage Service) is AWS object storage service. It is one of the most widely used AWS services due to its ability to handle various use cases, its scalability, data availability, performance, and security. For security, you can configure your bucket to be public, private, or grant specific users access to perform certain actions, either permanently or temporarily. This flexibility makes S3 suitable for a wide range of applications while ensuring data protection and access control.

S3 Presigned URL

When you create an S3 bucket, it is private by default, and it is up to you to change this setting based on your needs. If you want a user to upload or download files in a private bucket without making the bucket public or requiring AWS credentials or IAM permissions, you can create a presigned URL.

Presigned URLs work even if the bucket is public, but the main purpose of presigned URLs is to help you keep objects private while allowing limited and controlled access when necessary.

A presigned URL is an URL that grants temporary access to a specific object in an S3 bucket. It is generated for a specific object key within a specific S3 bucket, with an expiration time and a specified HTTP method (Get or Put).

Requirements for Generating Presigned URLs

A presigned URL must be generated by an AWS user or an AWS application that has access to the bucket and the object in the bucket at the time of creation. Generating a presigned URL does not require a call to the S3 service APIs. When a user makes an HTTP call with the presigned URL, AWS processes the request as if it was performed by the entity that generated the presigned URL.

Usage and Expiration

Presigned URLs can be shared with temporarily authorized users to allow them to download or upload objects. They can only be used for the method specified when generating the URL. For example, a GET presigned URL cannot be used for a PUT operation.

There is no default limit on the number of times a presigned URL can be used until it expires.

Get presigned URLs

Get presigned URLs are particularly useful for scenarios involving file sharing, secure download links and temporary access. For example, an educational platform can provide downloadable study materials to students using presigned URLs. A photographer can share high-resolution images with a client securely. A software company can distribute updates to its clients.

A GET presigned URL can be used directly in a browser or integrated into an application or webpage to download an object from an S3 bucket. It can be generated using the AWS Management Console, AWS CLI, or AWS SDK.

In the following, I will demonstrate how to generate a GET presigned URL using the AWS Management Console.

Generating Get presigned URL with the console

1. Choose the option “Share with a presigned URL” for your private object

2. Define the expiration time of the presigned URL

3. Access the private object using the presigned URL

Accessing a private object using object URL or presigned URL after expiration

Attempting to access a private object using its direct URL or a presigned URL after it has expired will result in an access denied message.

Put presigned URLs

Put Presigned URLs are commonly useful for scenarios where direct client-side uploads are beneficial. For instance, a social media platform can allow users to upload profile pictures directly to S3, bypassing the application server. An insurance company can enable clients to submit claim documents. A video-sharing platform can facilitate content creators in uploading their videos

When someone uses the Put presigned URL to upload an object, Amazon S3 creates the object in the specified bucket. If an object with the same key already exists in the bucket, Amazon S3 replaces the existing object with the newly uploaded object.

Put presigned URLs are typically generated using the AWS SDK. In the following section, I will show you how to generate a presigned URL using a Python script available in my GitHub repository and how to use it.

Generating Put presigned URL using python script

The Python script “generate_presigned_url.py” is available in my GitHub repository here.

Steps to Generate and Use a Presigned URL

1. Copy the Python script from my repository.
2. Run the following command to generate the presigned URL:

python generate_put_presigned_url.py your-bucket-name your-object-key --expiration 3600

3. Use the generated presigned URL to upload the file.

Ensure that the object key to upload and the bucket name where you need the upload match those used in the previous command to avoid errors. You can use tools like curl or Postman for this purpose.

curl -X PUT -T "/path/to/your-file.extension" "$PRESIGNED_URL"

A simple use case: React app file upload using presigned URL generated by lambda function

Simple React app file upload using presigned URL generated by lambda function

In this section, I show you how to set up a simple React application that allows users to upload files directly to an Amazon S3 bucket using Put presigned URLs generated by an AWS Lambda function. I used a public S3 bucket for hosting the frontend react code, which directly calls a lambda function via its URL to generate the presigned URL. This presigned URL is then used to upload files to a private S3 bucket.

Lambda function URL

The frontend calls the Lambda function using the Lambda function URL. A function URL is a dedicated HTTP(S) endpoint for your Lambda function. You can create and configure a function URL through the Lambda console or the Lambda API. Once created, the URL endpoint of the function URL never changes.

Note that, currently, function URLs are not supported in all regions, and they can only be accessed through the public Internet.

CORS

Cross-Origin Resource Sharing (CORS) defines a way for client web applications loaded in one domain to interact with resources in a different domain. To configure your bucket to allow cross-origin requests, you add a CORS configuration to the bucket.

A CORS configuration is a document that defines rules identifying the origins that are allowed to access your bucket, the HTTP methods supported for each origin, and other operation-specific information.

Steps to set up the simple app

A. Create the private S3 bucket

1. Create the private S3 bucket that will store the uploaded documents. This will be referred to as the “Documents bucket”.

2. Setup the CORS configuration of this bucket to allow interaction between «Frontend bucket» and «Documents bucket ». You can defer this step until after enabling the website feature for «Frontend bucket» if you want a more restrictive CORS policy.

B. Create lambda function

1. Create the lambda function that will generate the presigned URL for the files that we want to upload. I name it “put_presigned_URL_generator”

2. Add the Lambda function code from here

3. Update the lambda function role. Since the Lambda function generates the presigned URLs, the permissions assigned to the Lambda function will be those used by the presigned URL. Therefore, the Lambda role should include permissions for the put actions for "Documents bucket".

4. Go to the configuration section and enable the Function URL for your Lambda function with the authentication type of NONE

5. Add the name of the “Documents bucket” to the environment variables of your Lambda function.

C. Create the S3 frontend bucket

1. Create an S3 bucket to host the react app. I name it «Frontend bucket »

2. Change the settings of «Frontend bucket» to make it to public. This involves disabling block public access and updating the bucket policy to allow public access.

3. Clone the React application code from my GitHub repository here.

4. Replace the placeholder Lambda function URL in the code with your actual Lambda function URL.

5. Build your React app by running npm run build and upload the contents of the build folder to "Frontend bucket". You can use the following command:

aws s3 sync build/ s3://your-frontend-bucket-name

6. Enable the static website hosting feature for «Frontend bucket» and set the index.html file as the index document for the website

7. Access the bucket website endpoint to test the solution and ensure everything is working correctly.

To go further

This simple app provides a basic overview of how Lambda and S3 presigned URLs can be used for secure file uploads. However, for enhanced security and functionality, additional layers should be implemented to ensure that only authenticated users can generate presigned URLs.

I hope this guide helps you implement secure file management in your projects.