AWS S3 using LocalStack and Java

Where to start?

Howdy, fellow geeks, this is a guide on making simple upload files to S3 and downloading files from S3 programs in AWS SDK for Java and using LocalStack instead of aws directly. There is a lot going on here but it will be a piece of cake by the end of this guide. I have also added commonly faced problems by the end of this guide so if you face any problem just head over there. I will advise first understanding the concepts I am listing here and then coming back to continue with the guide

• Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html)
  -S3 buckets
  -S3 objects
• LocalStack (https://docs.localstack.cloud/getting-started/)
• AWS Java SDK (https://docs.aws.amazon.com/sdk-for-java/index.html)
• Docker (https://docs.docker.com/get-started/overview/)
  -What is Docker?
  -What are containers in docker?

Note: This is just a sample list of topics. You can cruise through the oceans of documentation as much as you like.

Overview:

Let’s first get an overview using a diagram and then we will break down the main problem into subproblems.

Source: https://localstack.cloud/

In this diagram, your application will be replaced by the application we will make in Java to create a bucket. Then the LocalStack will act as an imposter and will give your application the illusion of aws S3 service. We will run the LocalStack as a docker container.

Basic Application in Java:

Let’s create a basic application in Java. Here are the steps:

1-) First create a new project in Java. I am using OpenJDK 18. Select webapp as an archetype. (If you want you can deploy your application on apache server to send messages from postman and run the main on starting to poll messages)

2-)Make a basic folder hierarchy as shown in the screenshot below and add a file with any name.

3-) Now is the golden time for “Copy and paste”. Take the below code, I have copied myself from official documentation🙂
NOTE: You will need to read the explanation below the code snippet because your file path and filename would be different.

package service;

import com.amazonaws.AmazonServiceException;
import com.amazonaws.SdkClientException;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.CreateBucketRequest;
import com.amazonaws.services.s3.model.GetBucketLocationRequest;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.PutObjectRequest;
import java.io.File;


public class S3Service {
public static void main(String[] args) {
String bucketName = "example";
createBucket(bucketName);
System.out.println("Bucket has been created with name: " + bucketName);
 uploadFileToS3();
System.out.println("Files have been uploaded to the bucket");
downloadFilesFromS3();
System.out.println("Files have been downloaded from the bucket");
}


public static void createBucket(String bucketName)
{
try {
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(
"http://localhost:4566","us-east-1")).enablePathStyleAccess()
.build();
// if (!s3Client.doesBucketExistV2(bucketName)) {
// Because the CreateBucketRequest object doesn't specify a region, the
// bucket is created in the region specified in the client.
s3Client.createBucket(new CreateBucketRequest(bucketName,"us-east-1"));
// Verify that the bucket was created by retrieving it and checking its location.
String bucketLocation = s3Client.getBucketLocation(new GetBucketLocationRequest(bucketName));
System.out.println("Bucket location: " + bucketLocation);
// }
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it and returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
}


public static void uploadFileToS3()
{
String bucketName = "example";
String fileName = "examplefile.txt";
String filePath = "C:\\Users\\HotelKey\\Documents\\s3samplefiles\\" + fileName;
AmazonS3 client = AmazonS3ClientBuilder.standard().withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(
"http://localhost:4566",
"us-east-1")).enablePathStyleAccess().build();
PutObjectRequest request = new PutObjectRequest(bucketName,fileName,new File(filePath));
client.putObject(request);
}


public static void downloadFilesFromS3()
{
AmazonS3 client = AmazonS3ClientBuilder.standard().withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(
"http://localhost:4566",
"us-east-1")).enablePathStyleAccess().build();
String bucketName = "example";
String fileName = "examplefile.txt";
GetObjectRequest request = new GetObjectRequest(bucketName,fileName);
System.out.println(client.getObject(request).toString());
}
}

Let’s understand what is happening here because copying and pasting will not make you a good developer, right? (Did I say I have copied the code?)
In the S3Service class, we have 4 methods. One is main and the other 3 are used for creating buckets, uploading files to buckets and downloading files from buckets respectively.

main:

Every bucket should have a name just like variables, so I am creating a string “example” for the name of the bucket. Passing this string to the createBucket() method. Calling the other two methods for uploading and downloading the files. And printing some lines in between to see if everything is working fine.

createBucket:

To work with any aws service, you need to build a client in Java. In this case, we are creating a client named s3Client using AmazonS3ClientBuilder. You will notice that I have given an endpoint http://localhost:4566. If we were using aws directly, the endpoint would be the aws url but in our case, we are using LocalStack which is running in a docker container (Don’t worry we will set up this later). So our client is sending requests to LocalStack.

uploadFilesToS3:

You need the name of the bucket which we created earlier and the filepath of the file which we want to upload to S3(the imposter S3). Replace the file name as well as filepath with the file you want to upload.

downloadFilesFromS3:

You need the name of the bucket which we created earlier and the key of the file which we want to download from S3(the imposter S3). Replace the file name with the file you want to download.

4-) Now you must be thinking that I have skipped something because your Intellij is showing a lot of red lines so it’s time for tinkering with pom.xml

Add the following dependency in the pom.xml

<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.318</version>
</dependency>

If you don’t know how to add dependencies in the pom.xml, here is a screenshot:

Docker:

I am using windows so here is the guide on how to install docker:
https://docs.docker.com/desktop/install/windows-install/

There is a great chance that you will face problems, I have shared common problems in the end. If you have docker desktop installed, just open it using the start menu. We will use docker to create a container of LocalStack.

LocalStack:

There are multiple ways to set up LocalStack. Screenshot from the official documentation:

I would suggest LocalStackcli. Here is the link for details:
https://docs.localstack.cloud/getting-started/installation/#localstack-cli

If you are able to install LocalStack cli, you can verify using the command

If all goes well, open the docker in the background and type the following:

If you are getting this beautiful LocalStack logo, you are good to go to the next step.

AWS CLI:

We have made a java program to create buckets, upload and download items. But how can we make sure that the buckets are created? One way is to see the logs of LocalStack cli. That is great but what if we need to see all other details of the buckets like id etc. We can use AWS CLI for this purpose. All the work we have done here using Java can be done with AWS CLI with much simpler commands. We will use the list-buckets command to see the list of buckets that have been created.

How to install AWS CLI:

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

We will come back to aws cli later. Just install and verify if it has been installed correctly.

Maven Build:

Open pom.xml file in the application we built earlier. Maven builds a package right? In our example, we are not hosting it on apache server so replace ‘war’ with ‘jar’.

Because we will run our application through the main program, we also need to add maven compiler source and target to properly build the jar package. Add the following code snippet before your dependencies tag.

<properties>

<maven.compiler.source>18</maven.compiler.source>

<maven.compiler.target>18</maven.compiler.target>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

</properties>

Reload the pom.xml file and open the terminal. Type mvn clean install.

If you are getting the following message “BUILD SUCCESS”, you are the chosen one.

Now simply run the program. It will create a bucket, upload a file in it and then download the file. You can see if everything is working by seeing the output on your console or from the logs on LocalStack cmd.

IntelliJ output console:

Logs from LocalStack console: (I am getting GetBucket because I already created a bucket. You will get CreateBucket if you are running the program for the first time)

Great, isn’t it?

Debugging:

Open cmd and type the following line:

docker exec -it <container name> /bin/bash

Replace this <container name> with the id of your container.
→ Open the Containers tab on your docker desktop
→ The Containers tab will show the list of all the containers that are currently installed on your computer.
→ Click the copy button with the container id text to copy the id. (Our container is the LocalStack)
→ Replace the <container name> with the id and press enter.

This is what you will get.
Now we need to configure our aws cli with a profile. Enter the following command(Replace the name with yours)

aws configure — profile mughees

The first two are irrelevant because LocalStack does not verify credentials (when you will use aws cli with aws, you will need to add proper credentials). Add region “us-east-1” because this is the region we gave in our Java application. And finally give json as output format.

Add the following line to get the list of buckets. The endpoint is again the port at which LocalStack is running on docker.

aws s3api list-buckets — profile mughees — endpoint-url http://localhost:4566

You can see the list of buckets that you have created by your Java code. I already created a bucket with example2 that is why getting two buckets. You can also see the list of objects(files we uploaded) using the list-objects command. Just go to the documentation to see the list of all commands.

Hurray, you have made your first S3 application using AWS SDK for Java and LocalStack. Did someone say Ez? If you are frustrated because everything did not work according to plan, below is the list of common problems (took me hours to debug xd) you might have faced. I would recommend going through all the problems in order to be aware of common bugs you will be facing during your programming journey.

Common Problems:

Problem 1: As a newbie, you will name your package and class. And after copying the Java code, you will be wondering what’s the problem.
Solution:
Just make sure that the package name and the class name are the same.

Problem 2: As a smart developer, you refused to copy code from my code snippet and copied it from the official documentation. I am sharing the GitHub link:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/java/example_code/s3/src/main/java/aws/example/s3/CreateBucket.java
Now the problem is the official code does not take into account LocalStack, you will be frustrated in no time because you are doing everything correctly. Here is the issue, Java SDK provides different builders to build the client. You need to enable PathStyleAccess using the builder for the client. Otherwise, you will keep getting this error for the rest of your life.

Before hitting your head against the wall, here is how you can solve it.

Solution: If you are using AWS SDK 1.x, you will be using the AmazonS3ClientBuilder to build your client. Add a method enablePathStyleAccess()to enable PathStyleAccess. If you are using AWS SDK 2.x, you will be using S3Client to build your client. Just add a method .forcePathStyle(true)to enable PathStyleAccess. And your problem will be solved. I already added the method for your convenience in my code snippet.

Problem 3: I used a windows laptop and I faced a lot of problems while running docker. Here are some of them:

Problem: Docker Desktop shows “Docker is starting” and then automatically closes.

Possible Solutions: This can happen due to a lot of reasons. So there are multiple solutions:

Solution 1: Close your docker if it is already running from task manager and start it again.
Solution 2: Press ‘Windows + R’ to open the run window and enter ‘services.msc’.

You will get this window, click on any service and press l. Search for LxssManager.

The LxssManager server really is Windows Subsystem for Linux (or at least a major part of it). If it is disabled, enable it. Otherwise, restart it. If you are on an account other than administrator, you will need to start or restart the Lxss Manager for your user. Restart the docker and it will magically start running.

Solution 3: Reinstall your Ubuntu version. First uninstall the previous version and then reinstall using the command wsl — install -d Ubuntu-20.04 Any version you like