Upload and Retrieve files from AWS S3 using the pre-signed URL pattern in Java
In my journey to learn AWS, I stumbled upon the presigned URL pattern of S3. This pattern is very important for saving server bandwidth when dealing with uploading and downloading objects from the cloud.
Essentially, it works by bypassing the server entirely and freeing it from the burden of processing and waiting for upload/download time and saving crucial resources (see figure 1).
For the purpose of this tutorial, I will be using this simple blog project to illustrate how this works in Java. The project is a mock of a blog website with an Angular 8 client and Spring Boot backend with Postgres database and of course, S3 as a file system for saving “big” files, more details about the project and the complete code could be found on my gitHub.
The use-case that we will be considering is when a user tries to upload/update their profile picture or the article thumbnail. Following steps are taken:
1- client requests to upload/update a specific file.
2- server requests a presigned URL from S3 for that specific resource.
3- server sends back the URL to the client.
4- client makes a put/get a request with the resource itself to S3 directly.
let’s see how this works,
1- Setting an S3 bucket
you need to set up an S3 bucket and allow public access to it, this could be achieved by navigating to the permission tab after creating a new bucket and editing the CORS configuration. Paste the following snippet in the editor and hit save.
<?xml version=”1.0" encoding=”UTF-8"?>
<CORSConfiguration xmlns=”http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>keep in mind that this is by no means a violation of the security for your bucket, as no one will have any access to this bucket except your server. Through which you will have full control over how you would allow interacting with it and who would be granted access to which files in it.
2- Client Request to generate a presigned URL
the client can request two kinds of presigned URLs, one to “PUT” a new resource and one to “GET” an already persisted one.
Both are simple calls to a spring controller responsible for generating such URLs with the name of the file.
// for "PUT"
generatePresignedPutUrl(file: File): Observable<any> {return this.http.get('http://localhost:8080/api/articles/putImage/'
+ file.name, { responseType: 'text' });}// for "GET"
generatePresignedGetUrl(articleId: number): Observable<any>{return this.http.get('http://localhost:8080/api/articles/getImage/'
+ articleId, { responseType: 'text' });}
3- URL generation
As a first step, we need to import the AWS sdk to the project to be able to communicate with any AWS product. Import can be achieved by a maven dependency or Gradle. After that, we will have a method for generating both “PUT”, and “GET” urls.
public static String generatePresignedUrl
(String fileName, HttpMethod mehtod) {
try {
// Set the pre-signed URL to expire after 10 mins.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 10;
expiration.setTime(expTimeMillis);
// Generate the pre-signed URL
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(bucketName, fileName)
.withMethod(mehtod)
.withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(
generatePresignedUrlRequest);
logger.info("pre-signed URL for " + mehtod.toString()
+ " operation has been generated.");
return url.toString();
} catch (Exception e) {
e.printStackTrace();
}
logger.error("URL could not be generated");
return null;after generating the URLs, they are sent back to the client where they could be executed after attaching the needed file.
4.1 - Executing PUT requests
after getting the PUT url, the file could be attached directly to it and executed directly. I have created this helper function for that.
uploadfileToAWSS3(fileuploadurl: string, file: File):
Observable<any>{ const req = new HttpRequest(
'PUT',
fileuploadurl,
file,
{
reportProgress: true, // to track upload process
});
return this.http.request(req);
}
and of course, you can call this wherever you like in your code and subscribe to it and maybe check the response code to make sure it was a successful request.
4.2 — Executing GET requests
get request are easier to deal with, all that I‘ve done is binding the returned presigned url to the “src” attribute of the img tag in the HTML file.
// HTML
<img class="rounded" [src]=imageUrl *ngIf=imageUrl>// Typescript
imageUrl: string;
.....
getImage(): void {
this.userService.generatePresignedGetUrl(this.user.id)
.subscribe(imageUrl => {
this.imageUrl = imageUrl;
}),
(err: any) => {
console.log(err.error);
};
}
And that’s it. now the presigned URL is fully working. A very important Gotcha would be to watch your requests that are fired from inside the app and if they are containing any other authentication mechanism other than X-Amz-Algorithm from AWS.
This would happen if you are using JWT authentication headers attached to your requests. In this case, you would need to filter any S3 requests from this header.
I hope this would be helpful to anyone stumbling on such things. And if you have any improvements, corrections, or anything related to the code, comments are welcomed.
relevant resources
