Spring Boot for Dummies: Part 2.4 [AWS S3 & MongoDB]
In this series we will look at how to setup a MongoDB and Amazon S3 on cloud and connect out application to it. Here is link to part 2.3. So, let’s get started.
# Setting up MongoDB.
For a free Mongo DB instance go to MongoDB Atlas: Cloud Document Database | MongoDB. And create a free tier account with 512MB to 5GB of storage, Shared RAM and no credit card.
Here is how you can create a DB; we are basically creating a database by name of simolytodo and a user for our application simplytodo-app-user.
# Connecting to mongoDB
Add the following dependency to pom.xml.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>let’s add mongo config to our application-staging.yml , as I am trying to add this in staging profile.
spring:
datasource:
driverClassName: org.postgresql.Driver
url: jdbc:postgresql://localhost:5432/simplytodo
username: admin
password: mypassword
data:
mongodb:
uri: mongodb+srv://simplytodo-app-user:{MONGO_PASSWORD}@cluster0.cm64xq4.mongodb.net/simplytodo?retryWrites=true&w=majority
jpa:
show-sql: true
hibernate:
ddl-auto: updateHere, I have set MONGO_PASSWORD to my mongo password, and I have changed it form what I showed in the video (so don’t even try 😂). But doing this is not ideal and can cause security issues. We will see in future parts on how to use a key-vault to store this info and dynamically inject it at app runtime.
Try running the application now, it should run fine. make sure you replace {MONGO_PASSWORD} with your mongo DB password.
MongoDB is a document-based database, In a document-based database, data is stored in documents, which are similar to JSON objects and can contain nested fields and arrays. This allows for a flexible schema that can evolve as the application needs change. Unlike relational databases, not all documents in a collection are required to have the same fields, because document databases have a flexible schema.
Let’s update our TodoTask and its sub class like Metadata & DescriptionBlock to a @Document from @Entity
@Document(collection = "todo_task")
@Data
public class TodoTask {
@Id
private String id;
@NotEmpty(message = "Title cannot be null")
private String title;
private DescriptionBlock description; // description for this task, root node.
private TodoTaskStatus status = TodoTaskStatus.NOT_STARTED; // default value
@FutureOrPresent(message = "Due date cannot be in the past")
private Date dueDate; // due date for this task
Metadata metadata = new Metadata(); // metadata for this task
private Set<String> tags; // set of tags for this task
private long user_id; // user who owns this task
}
// DescriptionBlock
@Document(collection = "description_block")
@NoArgsConstructor
@AllArgsConstructor
@Builder
@Data
public class DescriptionBlock {
@Id
private String id = UUID.randomUUID().toString();
private DescriptionBlockType blockType;
private String content;
List<DescriptionBlock> childBlocks;
public void setBlockType(DescriptionBlockType blockType) {
this.blockType = blockType;
}
public void setContent(String content) {
if(this.blockType == null)
throw new IllegalStateException("Block type must be set before setting content");
this.content = content;
}
public void setChildBlocks(List<DescriptionBlock> childBlocks) {
if(this.blockType == null)
throw new IllegalStateException("Block type must be set before setting child blocks");
if(this.blockType == DescriptionBlockType.IMAGE)
throw new IllegalStateException("image block cannot have child blocks");
this.childBlocks = childBlocks;
}
public void addBlock(DescriptionBlockType blockType, String content) {
if(this.blockType == null)
throw new IllegalStateException("Block type must be set before adding child blocks");
this.childBlocks.add(DescriptionBlock.builder().blockType(blockType).content(content).build());
}
}
// Metadata
@Document(collection = "metadata")
@Data
public class Metadata<T> {
@Id
private String id = UUID.randomUUID().toString();
private Date createdAt = Date.from(java.time.Instant.now());
private Long created_by_user_id;
private Date modifiedAt = Date.from(java.time.Instant.now());
private String objectType;
}Also make sure to update User class
@Entity(name = "user")
@Table(name = "todo_user")
@Data
public class User {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@JsonIgnore
private Long id;
@NotEmpty(message = "Name cannot be empty")
private String name;
@Email
private String email;
@Password(message = "Invalid password. Password must be at least 8 characters long, contain an uppercase letter, a number, and a special character.")
@JsonProperty(access = JsonProperty.Access.WRITE_ONLY)
private String password;
@Phone(message = "Invalid phone number")
private String phone;
@Transient // this field will not be persisted in the database
private List<TodoTask> tasks; // as this is stored in DB now
}@Transient annotation marks the field to be not persisted in DB, to retrieve all tasks for a user, we can just search MongoDB for all tasks with particular User ID.
Note: In document-based DB we don’t need relations as such, but we may not want very large objects. In order to create a separate collection, we would need to add a repository for the class we want to create the collection for, and we will use the annotation
@DBref, signifying that it is a reference to another document. Spring Data MongoDB won’t create a new collection automatically just because it's being referenced. You need to ensure that the referenced documents are persisted separately, perhaps by saving them explicitly.
You may also need to disable @TodoTaskreposirity because we created that repo to use with relational DB’s.
If you don’t want to delete the code just do something like
@Profile("none") // bean will not be created as this profile is not there
@Deprecated // marks this depricated so that other programmers are aware.
@Repository
public interface TodoTaskRepository extends JpaRepository<TodoTask, Integer> {
// rest of code
}Let’s make a new repository and name it TodoTaskMongoRepo
@Repository
public interface TodoTaskMongoRepo extends MongoRepository<TodoTask, String> {
public TodoTask findById(int id);
public void deleteById(int id);
public TodoTask findByTitle(String title);
public TodoTask insert(TodoTask todoTask);
}See here we are extending MongoRepository and not JpaRepository. Some of the methods have also changed.
Let’s update our service to now save the new TodoTask object to MongoDB.
@Service
public class TodoService {
@Autowired
private User loggedUser; // fake user instance we created for ease in last part
@Autowired
private TodoTaskMongoRepo todoTaskRepository;
public TodoTask createOrUpdate(TodoTask todoTask) throws TodoException {
Metadata<User> metadata = new Metadata<>();
metadata.setCreated_by_user_id(loggedUser.getId());
metadata.setObjectType(User.class.getTypeName());
todoTask.setMetadata(metadata);
todoTask.setUser_id(loggedUser.getId());
return todoTaskRepository.insert(todoTask);
}
public TodoTask getTask(int id) throws TodoException {
return todoTaskRepository.findById(id);
}
public void delete(int id) throws TodoException {
todoTaskRepository.deleteById(id);
}
public List<TodoTask> getAllTasks() throws TodoException {
return todoTaskRepository.findAll();
}
}Now, if we try to create an object, we will see something like this.
here is request body for your reference:
{
"title": "This new and updated task",
"description": {
"blockType": "PLAIN_TEXT",
"content": "This is plain text description of new task",
"childBlocks": [
{
"blockType": "MD_FORMATTED",
"content": "#This is a heading \n This is its description",
"childBlocks": [
{
"blockType": "IMAGE",
"content": "https://th.bing.com/th/id/R.abb691e5453eda2fc5a9617bc7f07091?rik=zyzecjkT%2bXRqCA&pid=ImgRaw&r=0"
}
]
},
{
"blockType": "IMAGE",
"content": "https://th.bing.com/th/id/R.e5f7d4befcd9e717566c7271795ac189?rik=5D%2fjAfEuYEaZdw&riu=http%3a%2f%2fwww.pngall.com%2fwp-content%2fuploads%2f2016%2f03%2fSnoopy-Cartoon-PNG.png&ehk=QvBL8wC%2feLVIooFmMbZ%2f6GBsbIuRCBcC9yRzcvn3ckA%3d&risl=&pid=ImgRaw&r=0"
}
]
},
"tags": ["NewTask", "FirstTask"]
}
We can also verify that if the task is created in MongoDB, click Database > Browse Collections.
You can see that a new task has been created in MongoDB, you can now do HTTP GET all task to see, it is being retrieved properly.
You are now wondering, how we can write same complex queries as we did for relational databases.
Well, the annotation @Query still works, we just need to write a mongo query, and similar to spring JDBC template, we also have MongoTemplate . For obvious reasons, I cannot cover MQL (MongoDB Query Language) here, but here is a simple example.
// using @Query
@Query("{ 'user_id' : ?0 }")
public List<TodoTask> findAlltasksByUserId(long userId);For using Mongo Template, we need to create something similar to JDBC Template. Let's add a Mongo Template bean to DatabaseConfig
Configuration
public class DatabaseConfig {
// read uri from config
@Value("${spring.data.mongodb.uri}")
private String mongoURI;
// create beans
@Bean
public MongoClient mongoClient() {
return MongoClients.create(mongoURI);
}
@Bean
public MongoTemplate mongoTemplate() {
return new MongoTemplate(mongoClient(), "simplytodo");
}
// rest of code
}Now we can create a DAO class like TodoTaskMongoDAO
@Repository
public class TodoTaskMongoDAO {
private MongoTemplate mongoTemplate;
@Autowired
public TodoTaskMongoDAO(MongoTemplate mongoTemplate) {
this.mongoTemplate = mongoTemplate;
}
public List<TodoTask> getAllTasksbytitleForuser(long userId, String title){
Query query = Query.query(Criteria.where("user_id").is(userId).and("title").is(title));
return mongoTemplate.find(query, TodoTask.class);
}
// implement any custom query logic here
}Add a controller and service method (you should be able to do it now).
# Setting up Amazon S3
Go to aws.amazon.com (if that was not obvious already 😂) create account and follow these steps.
# Connecting to Amazon S3
You can start by adding the following dependency in pom.xml.
// dependencies for aws s3
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.21.32</version>
</dependency>Now we need to add S3 credentials to the `application-staging.yml`
spring:
datasource:
driverClassName: org.postgresql.Driver
url: jdbc:postgresql://localhost:5432/simplytodo
username: admin
password: mypassword
data:
mongodb:
uri: mongodb+srv://simplytodo-app-user:{MONGO_PASSWORD}@cluster0.cm64xq4.mongodb.net/simplytodo?retryWrites=true&w=majority
jpa:
show-sql: true
hibernate:
ddl-auto: update
aws:
access-key: {AWS_ACCESS_KEY}
secret-key: {AWS_SECRET_KEY}
region: us-east-2
s3:
bucket-name: simplytodo-bucket
bucket-url: https://simplytodo-bucket.s3.us-east-2.amazonaws.com # aws bucket urlAs you may have already guessed, we need to create a configuration like AmazonS3Config and AWSS3Service to be able to interact with aws s3.
@Configuration
public class AmazonS3Config {
@Value("${aws.access-key}")
private String accessKey;
@Value("${aws.secret-key}")
private String secretKey;
@Value("${aws.region}")
private String region;
@Bean
public S3Client s3Client() {
AwsBasicCredentials credentials = AwsBasicCredentials.create(accessKey, secretKey);
return S3Client.builder()
.region(Region.of(region))
.credentialsProvider(() -> credentials)
.build();
}
}And AWSS3Service (you can name it anything).
@Service
public class AWSS3Service {
@Autowired
private S3Client s3Client; // our s3 client bean
@Value("${aws.s3.bucket-name}")
private String bucketName; // bucket name -> we can also make this not a service and a regular class that takes in client and bucket name in ctor
public List<S3Object> listObjects() {
ListObjectsV2Response response = s3Client.listObjectsV2(builder -> builder.bucket(bucketName));
return response.contents();
}
public void uploadFileToS3(String key, MultipartFile file) throws IOException {
PutObjectRequest request = PutObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build();
// Upload file to S3
PutObjectResponse response = s3Client.putObject(request, RequestBody.fromInputStream(file.getInputStream(), file.getSize()));
// Handle response or perform other operations
}
public void deleteFileFromS3(String key) {
DeleteObjectRequest request = DeleteObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build();
// Delete file from S3
DeleteObjectResponse response = s3Client.deleteObject(request);
// Handle response or perform other operations
}
public byte[] downloadFileFromS3(String key) throws IOException {
GetObjectRequest request = GetObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build();
// Download file from S3
ResponseInputStream<GetObjectResponse> response = s3Client.getObject(request);
return response.readAllBytes();
}
}Note: here key in arguments is the key of aws resource in s3.
We just now create a controller and try to create a method that gets a sample image from s3.
First, upload a sample image on s3, i have uploaded an image named mongodb.png.
We just want to test if we are able to get this image from S3. I created a new controller and added a following endpoint.
@Controller
@RequestMapping("/api")
public class FileController {
@Autowired
private AWSS3Service awss3Service;
@GetMapping("/files/{fileName}")
public ResponseEntity<byte[]> getFile(@PathVariable String fileName) throws IOException {
var fileContent= awss3Service.downloadFileFromS3(fileName);
// Set appropriate headers for the file content
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.IMAGE_JPEG); // Adjust based on your file type
// Return the file content as part of the response body
return new ResponseEntity<>(fileContent, headers, HttpStatus.OK);
}
}Now, if we call this endpoint and ask for file-name mongodb.png it should get us that.
# Thinking about strategy
Let’s pause and think. We are storing the text and structure of content in MongoDB, and to store images and files we use AWS S3. We can then reference the key of file in S3 in mongo data. Here, we will store S3 object key in place of content where block type is IMAGE or FILE.
// mongo document obj
{
// data ..
"childBlocks": [
{
"blockType": "IMAGE",
"content": "https://th.bing.com/th/id/R.abb691e5453eda2fc5a9617bc7f07091?rik=zyzecjkT%2bXRqCA&pid=ImgRaw&r=0"
}
]
},
{
"blockType": "IMAGE",
"content": "https://th.bing.com/th/id/R.e5f7d4befcd9e717566c7271795ac189?rik=5D%2fjAfEuYEaZdw&riu=http%3a%2f%2fwww.pngall.com%2fwp-content%2fuploads%2f2016%2f03%2fSnoopy-Cartoon-PNG.png&ehk=QvBL8wC%2feLVIooFmMbZ%2f6GBsbIuRCBcC9yRzcvn3ckA%3d&risl=&pid=ImgRaw&r=0"
}
]
// other data ...
}
We are essentially just avoiding storing image and file data in MongoDB and storing it to S3 and we still need to reference to S3 data in mongo to make that connection (I hope that makes sense).
The problem, however, is that adding and deleting images and files like this is not very efficient. It requires more sophisticated architecture. Especially, if we also have a frontend web service that saves changes automatically as any change in task object detected. Also, task object can be very long and may not be advisable to update the whole task for a minor change.
Creating elegant solution to this is possible, and may require some caching, and reimagining the task as series of generic blocks, and fixing a block size, then saving the chunks/blocks of one task into database, determining the diff and updating only necessary blocks. Also, we can add some metadata to S3 files, so that files are not immediately deleted (can be scheduled to perform bulk deletions).
All this (for now) is out of scope of this part. And shall be covered in later parts about scaling our application. But it is important to think about such complex scenarios as it helps us better understand and make conscious decisions of our program architecture.
If you have any questions, please feel free to add a comment or highlight, I will try to reply as soon as possible. Thank you. Happy Coding :)
