Open in app

Sign in

Write

Sign in

AWS S3 with Java using Spring Boot

A functional tutorial, creating a Photo Album project.

Gustavo Miranda
Analytics Vidhya
9 min readMar 22, 2020

--

Spring + AWS S3 + Maker Project

Today I will show for you how to make this work, to do this you need:

  1. Intermediate knowledge with Spring Framework
  2. AWS Account
  3. Java 11+ and Maven
  4. MongoDB

I like to start from the Backend

Is more simple do it from the backend, for this, we need to create a Bucket on AWS S3. Buckets is an simple abstraction for us know “where are my files?”.

Preparing AWS S3

In your AWS Console go to Storage Session and click on S3:

AWS > Storage > S3

Click on “Create bucket”:

Amazon S3 — Buckets screen

Now, you need to inform basic information about your bucket, for you identify it after for future maintain (the bucket name is unique on all Amazon S3):

Create bucket form

Alright, now you have a bucket on AWS S3, now we need create a “Access Key” and “Secret Key” to access your bucket on AWS Java SDK. Back to the AWS Console and search “IAM on Security, Identity, & Compliance” group:

AWS > Security, Identity, & Compliance > IAM

Don’t scary with this panel, we don’t need to change anything here, go to “Users” menu and click on “Add user”:

Create user on AWS

Follow the steps:

Create user on AWS — Step 1 — User Informations

Now, on second step, you need to select “AmazonS3FullAccess” because this user will be add/remove images from your bucket.

Create user on AWS — Step 2 — Permissions

Step 3, “Add tags” is optional, so, go to Step 4, “Review” and “Create user”.

Right, now, on Step 5, AWS show for us your “Access key ID” and the “Secret access key”, copy and don’t forget this!

Create user on AWS — Step 5 — Access Key and Secret Key

Creating Spring Project

Consider you have an intermediate knowledge on Spring Framework, go to Spring Initializr and create your project.

Spring Initializr — Step 1 — Project Information

With these dependencies:

Spring Initializr — Step 2— Dependencies

Now, click on “Generate”.

Unzip your project and open on your favorite IDE, in my case IntelliJ.

First, we need to import the AWS Java SDK in the project, go to your “pom.xml” and add the dependency:

<!-- Maker, to create the API more faster -->
<dependency>
   <groupId>com.github.gustavovitor</groupId>
   <artifactId>maker-mongo</artifactId>
   <version>${maker-mongo.version}</version>
</dependency><!-- Amazon S3 -->
<dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>aws-java-sdk</artifactId>
   <version>${amazon-sdk.version}</version>
</dependency>

<!-- Apache Commons for FileUtils and misc. -->
<dependency>
   <groupId>commons-io</groupId>
   <artifactId>commons-io</artifactId>
   <version>${commons-io.version}</version>
</dependency>

Properties:

<properties>
    <java.version>11</java.version>
    <maker-mongo.version>0.0.6</maker-mongo.version>
    <amazon-sdk.version>1.11.728</amazon-sdk.version>
    <commons-io.version>2.6</commons-io.version>
</properties>

Alright, basicly AWS Java SDK works like a API, you need to make one call on AWS API and send yours files to bucket.

Add these properties on application.properties:

amazon.s3.bucket-name=awesome-project-java-flutter
amazon.s3.endpoint=https://awesome-project-java-flutter.s3-sa-east-1.amazonaws.com/
amazon.s3.access-key=${ACCESS_KEY}
amazon.s3.secret-key=${SECRET_KEY}

For more security, create two environment variables in your machine, ACCESS_KEY and SECRET_KEY and populate it with IAM informations.

Now, create a AmazonClientService:

package com.github.gustavovitor.photos.service.amazon;import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import javax.annotation.PostConstruct;

@Service
public class AmazonClientService {

    // AmazonS3 Client, in this object you have all AWS API calls about S3.
    private AmazonS3 amazonS3;

    // Your bucket URL, this URL is https://{bucket-name}.s3-{region}.amazonaws.com/
    // If you don't know if your URL is ok, send one file to your bucket using AWS and
    // click on them, the file URL contains your bucket URL.
    @Value("${amazon.s3.endpoint}")
    private String url;

    // Your bucket name.
    @Value("${amazon.s3.bucket-name}")
    private String bucketName;

    // The IAM access key.
    @Value("${amazon.s3.access-key}")
    private String accessKey;

    // The IAM secret key.
    @Value("${amazon.s3.secret-key}")
    private String secretKey;

    // Getters for parents.
    protected AmazonS3 getClient() {
        return amazonS3;
    }

    protected String getUrl() {
        return url;
    }

    protected String getBucketName() {
        return bucketName;
    }

    // This method are called after Spring starts AmazonClientService into your container.
    @PostConstruct
    private void init() {

        // Init your AmazonS3 credentials using BasicAWSCredentials.
        BasicAWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);

        // Start the client using AmazonS3ClientBuilder, here we goes to make a standard cliente, in the
        // region SA_EAST_1, and the basic credentials.
        this.amazonS3 = AmazonS3ClientBuilder.standard()
                .withRegion(Regions.SA_EAST_1)
                .withCredentials(new AWSStaticCredentialsProvider(credentials))
                .build();
    }

}

Okay, with this client you can extends them to use the online client.

Now, we need to save basic informations about the AWS Images, for do this, we goes to create a simple domain/document:

package com.github.gustavovitor.photos.domain.amazon;import lombok.Data;
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;

import javax.validation.constraints.NotNull;

@Data
@Document
public class AmazonImage {

    @Id
    private String amazonUserImageId;

    @NotNull
    private String imageUrl;

}

Following the Spring, create a Repository for it:

package com.github.gustavovitor.photos.repository.amazon;

import com.github.gustavovitor.photos.domain.amazon.AmazonImage;
import org.springframework.data.mongodb.repository.MongoRepository;

public interface AmazonImageRepository extends MongoRepository<AmazonImage, String> {
}

Spring provide for us one especificly Form Data called MultipartFile, but you can’t upload this type on your bucket, bucket need a java.io.File instead. So, for do this conversion, we need create a Util class like this:

package com.github.gustavovitor.photos.util;

import org.springframework.web.multipart.MultipartFile;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.Date;
import java.util.Objects;

public class FileUtils {

    public static File convertMultipartToFile(MultipartFile file) throws IOException {
        File convertedFile = new File(Objects.requireNonNull(file.getOriginalFilename()));
        FileOutputStream fileOutputStream = new FileOutputStream(convertedFile);
        fileOutputStream.write(file.getBytes());
        fileOutputStream.close();
        return convertedFile;
    }

    public static String generateFileName(MultipartFile multipartFile) {
        return new Date().getTime() + "-" + Objects.requireNonNull(multipartFile.getOriginalFilename()).replace(" ", "_");
    }

}

Awesome, now we can do a service to send images to bucket:

package com.github.gustavovitor.photos.service.amazon;

import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.DeleteObjectRequest;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.github.gustavovitor.photos.domain.amazon.AmazonImage;
import com.github.gustavovitor.photos.repository.amazon.AmazonImageRepository;
import com.github.gustavovitor.photos.service.amazon.except.FileConversionException;
import com.github.gustavovitor.photos.service.amazon.except.InvalidImageExtensionException;
import com.github.gustavovitor.photos.service.amazon.util.AmazonClientService;
import com.github.gustavovitor.photos.util.FileUtils;
import com.github.gustavovitor.util.MessageUtil;
import lombok.extern.log4j.Log4j2;
import org.apache.commons.io.FilenameUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

@Log4j2
@Service
public class AmazonS3ImageService extends AmazonClientService {

    @Autowired
    private AmazonImageRepository amazonImageRepository;

    // Upload a List of Images to AWS S3.
    public List<AmazonImage> insertImages(List<MultipartFile> images) {
        List<AmazonImage> amazonImages = new ArrayList<>();
        images.forEach(image -> amazonImages.add(uploadImageToAmazon(image)));
        return amazonImages;
    }

    // Upload image to AWS S3.
    public AmazonImage uploadImageToAmazon(MultipartFile multipartFile) {

        // Valid extensions array, like jpeg/jpg and png.
        List<String> validExtensions = Arrays.asList("jpeg", "jpg", "png");

        // Get extension of MultipartFile
        String extension = FilenameUtils.getExtension(multipartFile.getOriginalFilename());
        if (!validExtensions.contains(extension)) {
            // If file have a invalid extension, call an Exception.
            log.warn(MessageUtil.getMessage("invalid.image.extesion"));
            throw new InvalidImageExtensionException(validExtensions);
        } else {

            // Upload file to Amazon.
            String url = uploadMultipartFile(multipartFile);

            // Save image information on MongoDB and return them.
            AmazonImage amazonImage = new AmazonImage();
            amazonImage.setImageUrl(url);

            return amazonImageRepository.insert(amazonImage);
        }

    }

    public void removeImageFromAmazon(AmazonImage amazonImage) {
        String fileName = amazonImage.getImageUrl().substring(amazonImage.getImageUrl().lastIndexOf("/") + 1);
        getClient().deleteObject(new DeleteObjectRequest(getBucketName(), fileName));
        amazonImageRepository.delete(amazonImage);
    }

    // Make upload to Amazon.
    private String uploadMultipartFile(MultipartFile multipartFile) {
        String fileUrl;

        try {
            // Get the file from MultipartFile.
            File file = FileUtils.convertMultipartToFile(multipartFile);

            // Extract the file name.
            String fileName = FileUtils.generateFileName(multipartFile);

            // Upload file.
            uploadPublicFile(fileName, file);

            // Delete the file and get the File Url.
            file.delete();
            fileUrl = getUrl().concat(fileName);
        } catch (IOException e) {

            // If IOException on conversion or any file manipulation, call exception.
            log.warn(MessageUtil.getMessage("multipart.to.file.convert.except"), e);
            throw new FileConversionException();
        }

        return fileUrl;
    }

    // Send image to AmazonS3, if have any problems here, the image fragments are removed from amazon.
    // Font: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#putObject%28com.amazonaws.services.s3.model.PutObjectRequest%29
    private void uploadPublicFile(String fileName, File file) {
        getClient().putObject(new PutObjectRequest(getBucketName(), fileName, file)
                .withCannedAcl(CannedAccessControlList.PublicRead));
    }

}

This is my project right now:

Project Design Pattern

If you read this post for AWS S3, this is everything you need to know, after this, I will show for you codes about Maker Project and Flutter (Flutter in another post, this post is too long).

Building a Simple Photo Album API

This API is very simple, we need one API route to upload photos and other route to get information about these photos, right? So I want to use the Maker Project to do this for us.

Maker Project is an abstraction to speed up the development time of RestAPI using Spring Framework, who?

Every pardonized API needs a lot of CRUD’s (Create Read Update Delete) operations, and these CRUD’s can cost time, and time for me is too short. Thinking this, I created for my personal projects the Maker, with this, I can build an CRUD API in 10 minutes, with all business rules applied.

Using Maker Project you can focus all of your time in business rules.

So, let’s go to create the Album entity:

package com.github.gustavovitor.photos.domain.album;

import com.github.gustavovitor.photos.domain.amazon.AmazonImage;
import lombok.Data;
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;

import javax.validation.constraints.NotBlank;
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Size;
import java.util.List;

@Data
@Document
public class Album {

    @Id
    private String albumId;

    @Size(max = 64)
    @NotNull
    @NotBlank
    private String title;

    // A list of AmazonImage, for this simple project, the AmazonImage is one request
    // and AlgumPhoto is another request.
    private List<AmazonImage> images;

}

Okay, now we need create a SpecificationObject for the Album (Maker requirements):

package com.github.gustavovitor.photos.repository.album.spec;

import com.github.gustavovitor.maker.repository.MongoSpecificationBase;
import com.github.gustavovitor.photos.domain.album.Album;
import com.github.gustavovitor.photos.domain.album.QAlbum;
import com.querydsl.core.BooleanBuilder;
import com.querydsl.core.types.Predicate;

import javax.management.ReflectionException;

import static java.util.Objects.nonNull;

public class AlbumSpecification extends MongoSpecificationBase<Album> {
    public AlbumSpecification(Album object) throws ReflectionException {
        super(object);
    }

    @Override
    public Predicate toPredicate() {
        BooleanBuilder builder = new BooleanBuilder();
        if (nonNull(getObject().getTitle())) {
            builder.and(QAlbum.album.title.containsIgnoreCase(getObject().getTitle()));
        }
        return builder;
    }
}

Where is QAlbum.class? QAlbum is generated using one plugin to do this, add on your pom.xml, inside build plugins this:

<!-- Code Generation QueryDsl -->
<plugin>
   <groupId>com.mysema.maven</groupId>
   <artifactId>apt-maven-plugin</artifactId>
   <version>1.1.3</version>
   <dependencies>
      <dependency>
         <groupId>com.querydsl</groupId>
         <artifactId>querydsl-apt</artifactId>
         <version>4.2.2</version>
      </dependency>
   </dependencies>
   <executions>
      <execution>
         <phase>generate-sources</phase>
         <goals>
            <goal>process</goal>
         </goals>
         <configuration>
            <outputDirectory>target/generated-sources/apt</outputDirectory>
            <processor>
               org.springframework.data.mongodb.repository.support.MongoAnnotationProcessor
            </processor>
            <logOnlyOnError>true</logOnlyOnError>
         </configuration>
      </execution>
   </executions>
</plugin>

After, run the command “mvn clean install -DskipTests” to generate code.

Note: You need to specify to your IDE where is the generated source root, in my case, using IntelliJ, I need to right click on /target/generated-sources and Mark Directory As -> Generated Sources Root.

Now, we need create a repository:

package com.github.gustavovitor.photos.repository.album;

import com.github.gustavovitor.maker.repository.MongoRepositoryMaker;
import com.github.gustavovitor.photos.domain.album.Album;

public interface AlbumRepository extends MongoRepositoryMaker<Album, String> {
}

Note: extends MongoRepositoryMaker, using the Maker Project to do methods for us.

Alright, now, create a service:

package com.github.gustavovitor.photos.service.album;

import com.github.gustavovitor.maker.service.MongoServiceMaker;
import com.github.gustavovitor.photos.domain.album.Album;
import com.github.gustavovitor.photos.repository.album.AlbumRepository;
import com.github.gustavovitor.photos.repository.album.spec.AlbumSpecification;
import org.springframework.stereotype.Service;

@Service
public class AlbumService extends MongoServiceMaker<AlbumRepository, Album, String, Album, AlbumSpecification> {
    @Override
    public void beforeInsert(Album object) {
        // Business rules here.
    }

    @Override
    public void beforeUpdate(String objectId, Album object) {
        // Business rules here.
    }

    @Override
    public void beforePatch(Album object) {
        // Business rules here.
    }

    @Override
    public void beforeDelete(String objectId) {
        // Business rules here.
    }
}

Maker Project do for us a lot of methods, inside MongoServiceMaker you can see these methods.

Alright, now, we need to create the resources/endpoints:

package com.github.gustavovitor.photos.resource.album;

import com.github.gustavovitor.maker.resource.MongoResourceMaker;
import com.github.gustavovitor.photos.domain.album.Album;
import com.github.gustavovitor.photos.service.album.AlbumService;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/album")
public class AlbumResource extends MongoResourceMaker<AlbumService, Album, String, Album> {

}

Finally, one more resource/endpoint for Amazon:

package com.github.gustavovitor.photos.resource.amazon;

import com.github.gustavovitor.photos.domain.amazon.AmazonImage;
import com.github.gustavovitor.photos.service.amazon.AmazonS3ImageService;
import lombok.Getter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestPart;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import java.util.List;

@Getter
@RestController
@RequestMapping("/amazon")
public class AmazonResource {

    @Autowired
    private AmazonS3ImageService amazonS3ImageService;

    @PostMapping("/images")
    public ResponseEntity<List<AmazonImage>> insertImages(@RequestPart(value = "images") List<MultipartFile> images) {
        return ResponseEntity.ok(getAmazonS3ImageService().insertImages(images));
    }

}

Let’s Go Test-it?

First, configure your MongoDB access on application.properties like this:

spring.data.mongodb.host=172.17.0.2
spring.data.mongodb.database=photo

And start your Spring Application.

Note: if you get error about SecurityAutoConfiguration, exclude this class on your SpringBootApplication, this error are caused because Maker Project include Spring Security dependencies.

package com.github.gustavovitor.photos;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration;

@SpringBootApplication(exclude = SecurityAutoConfiguration.class)
public class PhotosApplication {

   public static void main(String[] args) {
      SpringApplication.run(PhotosApplication.class, args);
   }

}

First, come with me and test the AmazonResource, open your Postman or another application to call the API and call:

Amazon Resource response.

Awesome!

Happy! Happy! Happy!

Checkout your Amazon bucket now!

Oh my God!

Okay, okay, calm down! Now, you need to get the response of the request to make the Album request:

Insert album request.

What?! Post? 201 Created? What? Yeah! Maker do this for us, everything we need to do this CRUD, the Maker Project do for us.

You can do request for your API like:

Get all Album Filtered request.

For more information about Maker Project please read the documentation on GitHub!

See? Send images to AWS S3 is simple, and more faster with Maker!

If you have any question about this, comment, or send a private message on my LinkedIn.

Integrate this API with a Flutter project following this another lesson: https://medium.com/analytics-vidhya/creating-an-album-photo-application-using-flutter-java-and-aws-s3-1d421c432b0d

The code of this API: https://github.com/gustavovitor/photo-album

Programming
Spring Framework
AWS
Tutorial
Developer

--

--

Gustavo Miranda
Analytics Vidhya

Written by Gustavo Miranda

23 Followers
Writer for

Fullstack Engineer na Viddaa

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams