Weekly Webtips
Published in

Weekly Webtips

Upload files directly to S3 with pre-signed URL

Photo by Joshua Sortino on Unsplash

Whether it be monolith server or micro-services, handling file upload often comes with the expense of bandwidth and compute requirements. Delegating this complexity to a more robust, resilient & scalable technology is always a good choice.

This post will describe an approach to upload files directly to AWS S3 using pre-signed URL. The pre -signed URL will be generated via AWS lambda. For backend service, we will use Serverless framework with Nodejs runtime. For frontend, it will be plain HTML & JavaScript.

Prerequisites:

  • Working AWS account setup and configured Access-Key and Secret-Key. Quick Guide.
  • S3 bucket to upload file

We will start by building a backend service first. It will expose a HTTP API which will be consumed by frontend app to get the pre-signed URL.

File structure of backend service.

- handler.js
- serverless.yml
- package.json

Simple Lambda function

Let’s begin by writing a simple lambda function inside handler.js file. The getUploadUrl function will take some parameters; based on which it will output pre-signed URL and filename.

const getUploadUrl = async (event) => {
return {
statusCode: 200,
body: {
preSignedUrl: "...",
fileName: "..."
}
};
};
export { getUploadUrl };

Inputs to this lamda function would be

  • contentType: Content-Type of file to upload
  • ext: File Extension. eg: png, jpg, jpeg etc.
  • isPublic: true/false value determining whether file will be publicly accessible or not

To provide these inputs, we will make use of AWS API Gateway. Via POST method these inputs will be sent to getUploadUrl function.

Serverless Configuration

Lets write the serverless.yml file to connect lambda function with HTTP API. Make sure to insert <AWS_REGION>. Install serverless-bundle by running npm install --save-dev serverless-bundle . It uses the serverless-webpack plugin internally.

service: s3-file-uploadplugins:
- serverless-bundle
provider:
name: aws
runtime: nodejs14.x
region: <AWS_REGION>
stage: dev
functions:
upload_url:
handler: handler.generateUploadUrl # lamda function
events:
- http:
path: /upload-url
method: post
cors: true

Run sls deploy to deploy. It will output a URL containing "/upload-url" at the end. Sending POST request to that URL should output 200 status code with the following body.

{
preSignedUrl: "...",
fileName: "..."
}

Upload files to S3

Lets modify handler.js file to get pre-signed URL from S3. Instead of using the original file name, the code uses current timestamp (to make it random). Access control list(ACL) controls whether the file is publicly available or not. Expires field determines lifetime of the URL in seconds. It defaults to 900 seconds (15 minutes). It can also be made configurable by passing it via HTTP body.

const AWS = require('aws-sdk');
AWS.config.update({ region: '<AWS_REGION>' });
const s3 = new AWS.S3();
const getUploadUrl = async (event) => { let filename = Date.now(); // random file name
var s3Params = {
Bucket: 'AWS_BUCKET_NAME',
Key: `${filename}.${event.body.ext}`,
ContentType: event.body.contentType,
ACL: event.body.isPublic ? 'public-read' : null,
Expires: 600 // 10 minutes
};
const url = s3.getSignedUrl('putObject', s3Params);
return {
statusCode: 200,
body: {
preSignedUrl: url,
fileName: `https://<AWS_BUCKET_NAME>.s3.<AWS_REGION>.amazonaws.com/${filename}.${event.body.ext}`
}
};
};
export { getUploadUrl };

Serverless Permissions

As seen on the code above, it has to access S3 and change ACL. For that, serverless service needs to be given certain permissions. For now lets give it 3 permissions(get, put, put-ACL) on the specific S3 bucket. The modified serverless.yml file is given below.

# ...
# ...
name: aws
runtime: nodejs14.x
region: <AWS_REGION>
stage: dev
iamRoleStatements:
- Effect: Allow
Action:
- s3:GetObject
- s3:PutObject
- s3:PutObjectAcl
Resource: "arn:aws:s3:::<AWS_BUCKET_NAME>/*"
functions:
upload_url:
# ...
# ...

This point marks the completion of backend service. Sending post request to the service should give pre-signed URL and filename as shown in example below. Try it on Postman or something similar.

Request

{
"contentType": "image/jpeg",
"ext": "jpg",
"isPublic": false
}

Sample Response

{
"url": "https://<AWS_BUCKET_NAME>.s3.<AWS_REGION>.amazonaws.com/1634026320180.jpg?AWSAccessKeyId=XXXXXXXXXX&Content-Type=image%2Fjpeg&Expires=1634027220&Signature=e1Gfy1hSzaPO9FfUrd8vSwz48bY%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEIj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQC3ClLmXAmeAfY4C4FcfY7V4fElb214mt%2FbKO%2FPzMonawIhANGNZmI9iSc28iQ2oEgMM%2BGlYsGOpkBUMaMbfQ3%2Fa8O07kGLe8boHpAgZmbyARvkTvoOSb3QBpMKP2HNnMcj4KvGZ%2FLNhoPVZdd870Mul%2Fq5AJvkpqEqyxlUdRxkh0Q%2FC1dgpixKlSXYnxjVU7n%2Fy7vBxWTBk8MMpiAeEJtFjxPStf261BL9NuvG2TZFfk%2Bk04LF%2FOb8H15vfXSmUD0WYoi3MYNnI15RusKx0RESmN4",
"fileUrl": "https://<AWS_BUCKET_NAME>.s3.<AWS_REGION>.amazonaws.com/1634026320180.jpg"
}

Frontend

Its time to build a simple web app to consume this API. Let’s use the API endpoint to actually get the pre-signed URL and upload file. We will start with a simple html page with file upload input, a text field to set lamda URL, label to show file upload progress and an image preview.

<input type="text" id="url" placeholder="Input Lamda URL">
<br><br>
<input type="file" id="fileInput">
<p>Upload percentage: <span id="uploadPercent"></span> </p>
<br><br>
<img src="" id="imgPreview" height="400" width="600" />

JavaScript code is set to execute when file is available to browser. It then extracts Content-Type and file extension of uploaded file and calls the specified api to get pre-signed URL. Again another PUT request is sent to pre-signed URL including uploaded file as HTTP Body. To keep things simple, axios is used for making HTTP calls. Code given below is self explanatory.

document.getElementById("fileInput").addEventListener('change', handleFileChange);async function handleFileChange(e) {  const URL = document.getElementById("url").value;  try {    // take the first file from FilesList
const file = e.target.files[0];
// Extract Content-Type & filename
const {
type,
name
} = file;
// Get upload URL
let resp = await axios.post(
URL, {
contentType: type,
ext: name.split(".").pop(), // take only file extension
isPublic: true, // make it public so you can preview
}
);
// Upload at URL
const {
url,
fileUrl
} = resp.data;
await axios.put(url, file, {
headers: {
"Content-Type": file.type
},
onUploadProgress: (e) => {
// Show progress
var percentCompleted = Math.round((e.loaded * 100) / e.total);
document.getElementById("uploadPercent").innerHTML = percentCompleted;
},
});
// suuccessfully uploaded
document.getElementById("uploadPercent").innerHTML = "Completed";
document.getElementById("imgPreview").setAttribute("src", fileUrl);
} catch (e) { // Log errors
console.log(e);
}
}

Demo

Copy and paste your API Gateway’s URL into the demo below, and upload an image. It should upload the image to your S3 bucket which you can check from AWS console.

http://jsfiddle.net/vhwrgy0x/15/embedded/result/dark

https://jsfiddle.net/bibhuti/vhwrgy0x/15/

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store