Spring Boot with AWS S3 Bucket from zero to useful

A practical implementation for creating and reading .txt files in the AWS cloud

Hello, today I will leave a summary about the Amazon S3 service and how to use it in combination with Spring Boot. The idea is to make a presentation about the service and some possibilities of use with practical examples.

A summary of Amazon s3

Amazon Simple Storage Service is an infinite-scale object storage service.

When talking about Amazon S3 there are some concepts:

• Buckets: These are directories and have a globally unique name
• Objects: These are files that have a key and this key is the full path. For example s3://my-bucket/my-file.txt

The maximum size of an object is 5TB and if the upload is larger than 5GB the multipart upload must be used.

• Versioning: Enabled at the bucket level. The same key overwrites and increments the version: 1, 2, 3… It’s the best practice to version your files.
  Under “Properties” of the bucket, you can enable this.
• S3 can also maintain static websites and make them available on the internet. If the return is an HTTP 403 (forbidden), it’s good to look at the policy and make sure it allows public access.

S3 Storage Classes

Amazon S3 Standard

• This class is used for general purposes;
• 99.999999999% object resiliency across multiple Availability Zones;
• Used for data that will be accessed frequently.

Amazon S3 Standard — Infrequent Access

• Used for data that has a lower frequency of access but requires fast access when needed;
• 99.99% availability;
• Lower cost compared to S3 Standard;
• Used for disaster recovery backups.

Amazon S3 One Zone

• Used for infrequently accessed data;
• It has a durability of 99.999999999% in a single AZ (availability zone);
• Used to store a secondary backup of copies of on-premises data, or data that can be recreated.

Amazon S3 Glacier Instant Retrieval

• Class used to archive things, so it has a low cost;
• Data storage here must be at least 90 days long;
• Millisecond recovery is great for data accessed once a quarter.

Amazon S3 Glacier Flexible Retrieval

• It’s also a class used to archive things, so it’s low cost;
• Data storage here must also be at least 90 days long;
• Suitable for data that can be accessed 1 or 2 times a year and retrieved asynchronously;
• Recovery times from minutes to hours.

Amazon S3 Glacier Deep Archive

• It has a data recovery time of up to 12 hours, being suitable for data that can be accessed 1 or 2 times a year;
• Provides 99.999999999% object resiliency across multiple Availability Zones;
• Data recovery can take up to 12 hours.

Amazon S3 Intelligent Tiering

This is a very interesting case. This storage class automatically reduces storage costs by automatically moving data to the most cost-effective tier based on the frequency of access.

For example, an object can be moved to a low access layer, for reasons of low access in everyday life. If this object is later accessed, it will be moved back to the frequently accessed tier.

• The durability of 99.999999999% of objects in multiple Availability Zones;
• It has a small monthly charge for monitoring and automatic levels;

Creating an S3 via the AWS Console

It’s time to create a bucket and it’s very simple, just search for “s3” and then click on “Create Bucket”.

Some data is required and the name field must be unique across AWS, not just your account. This field is flagged below:

About the versioning that I mentioned above, in the creation it is possible to activate:

Another way to create a bucket is using the AWS SDK (Software Development Kit) which provides tools for creating resources using programs. I will show below some simple ways to use Java and Spring Boot.

Generating access credentials

The first step is to create security credentials that will be used to access AWS services.

For that, look for the IAM service, and in users (don’t do it with root) use the option “Security credentials”

Setting up a Spring Boot project

Here I will create a project that enables the creation and maintenance of .txt files on Amazon s3.

I started by putting the AWS dependency on the project:

<dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>aws-java-sdk</artifactId>
   <version>1.11.163</version>
</dependency>

With this, we will have access to a series of classes and methods to interact with AWS services.

I’ll start configuring a client for access by creating a configuration class like this:

@Configuration
public class AWSConfig {

    public AWSCredentials credentials() {
        AWSCredentials credentials = new BasicAWSCredentials(
                "accesskey",
                "secretKey"
        );
        return credentials;
    }

    @Bean
    public AmazonS3 amazonS3() {
        AmazonS3 s3client = AmazonS3ClientBuilder
                .standard()
                .withCredentials(new AWSStaticCredentialsProvider(credentials()))
                .withRegion(Regions.US_EAST_1)
                .build();
        return s3client;
    }

}

In “accesskey” and “secretkey” each user will put their security credentials generated there in IAM.

After doing this, some actions are possible and I created a service class for them.

Bucket-level actions:

Create a bucket:

public void createS3Bucket(String bucketName) {
    if(amazonS3Client.doesBucketExist(bucketName)) {
        log.info("Bucket name already in use. Try another name.");
        return;
    }
    amazonS3Client.createBucket(bucketName);
}

List buckets:

public List<Bucket> listBuckets(){
    return amazonS3Client.listBuckets();
}

Delete a bucket:

public void deleteBucket(String bucketName){
    try {
        amazonS3Client.deleteBucket(bucketName);
    } catch (AmazonServiceException e) {
        log.error(e.getErrorMessage());
        return;
    }
}

Object-level actions:

Put an object in a bucket:

public void putObject(String bucketName, BucketObjectRepresentaion representation, boolean publicObject) throws IOException {

    String objectName = representation.getObjectName();
    String objectValue = representation.getText();

    File file = new File("." + File.separator + objectName);
    FileWriter fileWriter = new FileWriter(file, false);
    PrintWriter printWriter = new PrintWriter(fileWriter);
    printWriter.println(objectValue);
    printWriter.flush();
    printWriter.close();

    try {
        var putObjectRequest = new PutObjectRequest(bucketName, objectName, file).withCannedAcl(CannedAccessControlList.PublicRead);
        amazonS3Client.putObject(putObjectRequest);
    } catch (Exception e){
        log.error("Some error has ocurred.");
    }

}

List all objects' names:

public List<S3ObjectSummary> listObjects(String bucketName){
    ObjectListing objectListing = amazonS3Client.listObjects(bucketName);
    return objectListing.getObjectSummaries();
}

Make a download of an object:

public void downloadObject(String bucketName, String objectName){
    S3Object s3object = amazonS3Client.getObject(bucketName, objectName);
    S3ObjectInputStream inputStream = s3object.getObjectContent();
    try {
        FileUtils.copyInputStreamToFile(inputStream, new File("." + File.separator + objectName));
    } catch (IOException e) {
        log.error(e.getMessage());
    }
}

For the above download I used an Apache dependency that makes I/O much easier:

Delete an object:

public void deleteObject(String bucketName, String objectName){
    amazonS3Client.deleteObject(bucketName, objectName);
}

Delete multiple objects:

public void deleteMultipleObjects(String bucketName, List<String> objects){
    DeleteObjectsRequest delObjectsRequests = new DeleteObjectsRequest(bucketName)
            .withKeys(objects.toArray(new String[0]));
    amazonS3Client.deleteObjects(delObjectsRequests);
}

Moving an object between two buckets:

public void moveObject(String bucketSourceName, String objectName, String bucketTargetName){
    amazonS3Client.copyObject(
            bucketSourceName,
            objectName,
            bucketTargetName,
            objectName
    );
}

Finally, I created a controller to test this code:

@RestController
@RequestMapping(value = "/buckets/")
@RequiredArgsConstructor
public class ControllerTests {

    private final S3Service s3Service;

    @PostMapping(value = "/{bucketName}")
    public void createBucket(@PathVariable String bucketName){
        s3Service.createS3Bucket(bucketName);
    }

    @GetMapping
    public List<String> listBuckets(){
        var buckets = s3Service.listBuckets();
        var names = buckets.stream().map(Bucket::getName).collect(Collectors.toList());
        return names;
    }

    @DeleteMapping(value = "/{bucketName}")
    public void deleteBucket(@PathVariable String bucketName){
        s3Service.deleteBucket(bucketName);
    }

    @PostMapping(value = "/{bucketName}/objects")
    public void createObject(@PathVariable String bucketName, @RequestBody BucketObjectRepresentaion representaion) throws IOException {
        s3Service.putObject(bucketName, representaion);
    }

    @GetMapping(value = "/{bucketName}/objects/{objectName}")
    public File downloadObject(@PathVariable String bucketName, @PathVariable String objectName) throws IOException {
        s3Service.downloadObject(bucketName, objectName);
        return new File("./" + objectName);
    }

    @PatchMapping(value = "/{bucketName}/objects/{objectName}/{bucketSource}")
    public void moveObject(@PathVariable String bucketName, @PathVariable String objectName, @PathVariable String bucketSource) throws IOException {
        s3Service.moveObject(bucketName, objectName, bucketSource);
    }

}

Tests
So far we have the following operations:

POST: http://localhost:8080/buckets/bucket-name to create a bucket.

DELETE: http://localhost:8080/buckets/bucket-name to delete a bucket.

GET: http://localhost:8080/buckets/ to list all buckets.

POST: http://localhost:8080/buckets/bucket-name/objects with the following body to create an object:

{
    "objectName": "object-name.txt",
    "text": "value of object"
}

GET: http://localhost:8080/buckets/bucket-name/objects/object-name.txt to fetch an object by name and download it.

GET: http://localhost:8080/buckets/bucket-name/objects/ to list existing objects.

DELETE: http://localhost:8080/buckets/bucket-name/objects/object-name to delete an object.

DELETE: http://localhost:8080/buckets/bucket-name/objects/ with the following body to delete multiple objects at once:

["nome-objeto-1.txt", "nome-objeto-2.txt"]

PATCH: http://localhost:8080/buckets/bucket-name/objects/object-name.txt/bucket-name2 to move an object between two buckets.

Public vs Private

By now you may have noticed that both the bucket and the objects were created with public access, right?

But it is possible to create them with private access. The CannedAccessControlList enum can be used for this:

public enum CannedAccessControlList {
    Private("private"),
    PublicRead("public-read"),
    PublicReadWrite("public-read-write"),
    AuthenticatedRead("authenticated-read"),
    LogDeliveryWrite("log-delivery-write"),
    BucketOwnerRead("bucket-owner-read"),
    BucketOwnerFullControl("bucket-owner-full-control"),
    AwsExecRead("aws-exec-read");

    private final String cannedAclHeader;

    private CannedAccessControlList(String cannedAclHeader) {
        this.cannedAclHeader = cannedAclHeader;
    }

    public String toString() {
        return this.cannedAclHeader;
    }
}

Here I will just use public and private and the implementation could look like this:

public void putObject(String bucketName, BucketObjectRepresentaion representation, boolean publicObject) throws IOException {

    String objectName = representation.getObjectName();
    String objectValue = representation.getText();

    File file = new File("." + File.separator + objectName);
    FileWriter fileWriter = new FileWriter(file, false);
    PrintWriter printWriter = new PrintWriter(fileWriter);
    printWriter.println(objectValue);
    printWriter.flush();
    printWriter.close();

    try {
        if(publicObject) {
            var putObjectRequest = new PutObjectRequest(bucketName, objectName, file).withCannedAcl(CannedAccessControlList.PublicRead);
            amazonS3Client.putObject(putObjectRequest);
        } else {
            var putObjectRequest = new PutObjectRequest(bucketName, objectName, file).withCannedAcl(CannedAccessControlList.Private);
            amazonS3Client.putObject(putObjectRequest);
        }
    } catch (Exception e){
        log.error("Some error has ocurred.");
    }

}

With that I also changed the controller for tests:

@PostMapping(value = "/{bucketName}")
public void createBucket(@PathVariable String bucketName, @RequestParam boolean publicBucket){
    s3Service.createS3Bucket(bucketName, publicBucket);
}

And with that, I can choose to create an object with public or private access. I did the same thing with creating the buckets:

public void putObject(String bucketName, BucketObjectRepresentaion representation, boolean publicObject) throws IOException {

    String objectName = representation.getObjectName();
    String objectValue = representation.getText();

    File file = new File("." + File.separator + objectName);
    FileWriter fileWriter = new FileWriter(file, false);
    PrintWriter printWriter = new PrintWriter(fileWriter);
    printWriter.println(objectValue);
    printWriter.flush();
    printWriter.close();

    try {
        if(publicObject) {
            var putObjectRequest = new PutObjectRequest(bucketName, objectName, file).withCannedAcl(CannedAccessControlList.PublicRead);
            amazonS3Client.putObject(putObjectRequest);
        } else {
            var putObjectRequest = new PutObjectRequest(bucketName, objectName, file).withCannedAcl(CannedAccessControlList.Private);
            amazonS3Client.putObject(putObjectRequest);
        }
    } catch (Exception e){
        log.error("Some error has ocurred.");
    }

}

And the controller for tests:

@PostMapping(value = "/{bucketName}/objects")
public void createObject(@PathVariable String bucketName, @RequestBody BucketObjectRepresentaion representaion, @RequestParam boolean publicObject) throws IOException {
    s3Service.putObject(bucketName, representaion, publicObject);
}

And the requests look like this:

POST: http://localhost:8080/buckets/bucket-name/objects?publicObject=true with the following body to create an object:

{
    "objectName": "object-name.txt",
        "text": "value of object"
}

POST: http://localhost:8080/buckets/bucket-name?publicBucket=true to create a bucket.

This code is a proof of concept that is functional and available for reference here: https://github.com/mmarcosab/s3-example

Top 10 Spring MVC + REST Interview Questions Answers for Beginners and Experienced Developers

25 Essential skills Java Programmers and Architects can learn in 2022

The 2022 Java Programmer RoadMap