Building a CRUD Application with a NoSQL Database

A simple CRUD application with MongoDB running on a Docker container.

One of the many interesting points that arose while learning about Amazon DynamoDB during AWS Cloud Practitioner was that the use of NoSQL was a better choice than SQL for use cases where simple queries were applied to lookup tables. NoSQL offers:

1. schema-less or flexible schema models, making it easier to store varying types of data without the need for predefined schemas.
2. better performance for certain use cases, especially when dealing with large volumes of data or high throughput, due to their simpler data models and optimized query patterns.
3. excel at storing complex, nested data structures, making them suitable for storing semi-structured data like JSON documents.

My Donor Database is a simple CRUD application that I created to learn about NoSQL and compare its usage to SQL.

Scenario

Our team organizes quarterly blood donation drives, during which we track the event attendees (referred to as donors). If they are first-time attendees, we create a new entry with their details in Google Sheets. Otherwise, we indicate their attendance at the current drive. After each event, we analyze various metrics, including the total number of attendees, the breakdown between first-time and repeat donors, the sources through which attendees found out about the event, the geographic distribution of attendees (within and outside the estate), and their age breakdown.

Key Requirements

1. Add new donors to our database. Duplicate entries are not allowed
2. Lookup donor by name or contact number
3. Update count
4. Update contact details
5. Record the dates of blood donation events attended by donors, facilitating analysis of their participation patterns.
6. Record the medium through which donors learned about the event, aiding in refining publicity strategies effectively.

Defining the Donor Object

The donor object would consist of the following fields:

1. Contact Number: A unique identifier allowing easy communication with donors and ensuring accurate record-keeping.
2. Name: The donor’s name
3. Postal Code: Providing geographical information helps in analyzing the distribution of donors
4. Date of Birth: Used to track age
5. Count: Number of times a donor has attended the events

Project Setup

A step-by-step guide on how to set up MongoDB on Docker with IntelliJ can be found in my other article.

Add the required MongoDB dependency to your pom.xml file. This dependency will enable Spring Boot to integrate with MongoDB and provide support for MongoDB repositories.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>

The DonorController exposes the APIs, passing the request object to the DonorService. The DonorService processes the donor object before forwarding it to the DonorRepo to execute the database operations.

Project Structure

DB Operations

Here, I’ll delve into the implementation details of the database operations necessary to fulfil the requirements.

Donor Repo Setup

To set up the donor repository, begin by extending the MongoRepository interface, which provides out-of-the-box database operations.

public interface DonorRepo extends MongoRepository<Donor, String> {

}

Create New Donor

Utilize the insert or save methods provided by the interface to add the new donor object to the MongoDB database.

public Donor addDonor(Donor d) {
        d.setCount(INITIAL_COUNT);
        List<LocalDate> list = new ArrayList<>();
        list.add(DonorUtil.getDate());
        d.setDates(list);
        return donorRepo.insert(d);
    }

Handle Duplicate Entries

In SQL databases, we typically define one of the fields as the primary key to ensure no duplicate entries. However, in MongoDB, we can achieve a similar result using the @Indexed annotation. Simply annotate the desired field in your MongoDB document class and set auto-index-creation to true for automatic index creation, enforcing uniqueness constraints within the collection.

@Data
@Document
public class Donor {

    @Indexed(unique = true)
    private String contactNo;

}

spring.data.mongodb.auto-index-creation=true

When attempting to add a duplicate contact, MongoDB will raise the following exception: com.mongodb.MongoWriteException: E11000 duplicate key error collection

Donor Lookup by Name or Contact

In the DonorRepo interface, we define a method annotated with @Query to specify the query to be used. The ?0 placeholder is used to refer to a variable that is passed to the method. This method will be called from the service class.

public interface DonorRepo extends MongoRepository<Donor, String> {

    @Query("{ $or: [ {contactNo: '?0'} , {name: '?0'} ] }")
    Donor findDonor(String value);

}

This query performs a search against the contactNo and name fields, looking for a match with the provided value. This operation is akin to an SQL query using SELECT * WHERE contactNo=? OR name = ?

Update Count

There are 2 approaches to update the value of an existing object

(1) Save
Here, we retrieve the object, change the values and use the save method to insert it back into the database. This is a poor approach as it could result in concurrency issues, unnecessary reads and many more.

public Donor updateCount(String contactOrName) throws CustomException {
        Donor d = findDonor(contactOrName);
        int count = d.getCount();
        d.setCount(count+1);
        return donorRepo.save(d);
    }

(2) MongoTemplate
Utilizing MongoTemplate allows for changes to be made directly on the database side, eliminating the need for manual editing and saving. We define an interface and provide its implementation.

public interface CustomRepo {

    void updateCount(String nameOrContact) throws CustomException;

}

@Repository
public class CustomRepoImpl implements CustomRepo {

    @Autowired
    MongoTemplate mongoTemplate;

    @Override
    public void updateCount(String nameOrContact) throws CustomException {

        Criteria criteria = new Criteria().orOperator // (1) 
                (
                        Criteria.where("name").is(nameOrContact),
                        Criteria.where("contactNo").is(nameOrContact)
                );
        Query query = new Query(criteria); // (2)
        Update update = new Update(); // (3)
        update.inc("count", 1);
        update.push("dates", DonorUtil.getDate());
        // (4)
        UpdateResult ur = mongoTemplate.updateFirst(query, update, Donor.class);

        if (ur.getModifiedCount() == 0) { // (5)
            throw new CustomException("update count failed");
        }

    }

}

1. Define criteria for update: when either contact number or name is equal to the given value
2. Create a query with the defined criteria
3. Define the update operation: increment the count by 1 and push the current date to the ‘dates’ array. $inc is used to increment the existing value by the specified value
4. Execute the update operation using the MongoTemplate
5. If no document was updated, throw a custom exception

Update Contact Number

This follows the same steps as we did to update the count. However, instead of using the $inc operator, we use the $set operator to set the new contact number.

 @Override
    public void updateContact(String nameOrContact, String newContactNo) throws CustomException {

        Criteria criteria = new Criteria().orOperator(
                Criteria.where("name").is(nameOrContact),
                Criteria.where("contactNo").is(nameOrContact)
        );

        Query query = new Query(criteria);

        Update update = new Update();
        update.set("contactNo", newContactNo);

        UpdateResult updateResult = mongoTemplate.updateFirst(query, update, Donor.class);

        if(updateResult.getModifiedCount() == 0){
            throw new CustomException("update contact number failed");
        }

    }

Date & Publicity Medium Record

To record the dates of blood donation events attended by donors, I included a Map<LocalDateTime,String> in the Donor object. This map field is updated in two scenarios:

(1) When a new donor is added to the database
A new map is initialized, and the date based on the timestamp and corresponding publicity medium is added to the list.

public Donor addDonor(Donor d) {
        d.setCount(INITIAL_COUNT);
        List<LocalDate> list = new ArrayList<>();
        list.add(DonorUtil.getDate());
        d.setDates(list);
        return donorRepo.insert(d);
    }

(2) When the count is updated
The push method of the MongoTemplate is used to add the timestamp date to the existing listupdate.push("dates", DonorUtil.getDate())

Retrieving the local date and time is handled in two separate methods. To ensure consistency in the format used by both, I’ve extracted it into a separate method getDate().

Recording the publicity medium follows a similar approach but with a List<String>.

I tested another approach where I used a instead of 2 separate lists. However, this object cannot be inserted into the MongoDB as it will not accept a key value that contains dots. The exception thrown was org.springframework.data.mapping.MappingException: Map key 2024–03–14T19:29:01.50org.springframework.data.mapping.MappingException: Map key 2024–03–14T19:29:01.401372400 contains dots but no replacement was configured! Make sure map keys don’t contain dots in the first place or configure an appropriate replacement!

Observations

The Inefficiency of Using Lists for Tracking

The issue with using 2 separate lists to track date and publicity medium is that we need to correlate them on our own. For example, the 3rd data in the publicity medium defines how the donor found out about the event for the 3rd data in the dates list. This will result in complex processing.

Instead, I use a map with key = date and value = publicity Map<LocalDateTime,String> . This threw the error org.springframework.data.mapping.MappingException: Map key 2024–03–14T19:29:01.502345 contains dots but no replacement was configured .

Keys in MongoDB maps cannot contain dots. As a fix, we use a Date Formatter to format the date to yyyy-MM-dd HH:mm:ss and remove the dots in the keys

Handling Duplicate Names

In the scenario where there is a duplicate name and we search using the name (instead of their contact), a result cannot be returned: org.springframework.dao.IncorrectResultSizeDataAccessException

Cause: The find query method is expecting a single Donor object but we are getting a list of objects

@Query("{ $or: [ {contactNo: '?0'} , {name: '?0'} ] }")
    Donor findDonor(String value);

Solution: Since it is a valid scenario where people might have the same names, the method must return a list instead of a single donor object

Value of Flexible Schemas

I’ve come to appreciate the value of flexible schemas. Initially, I designed the Donor object with a Dates list. As an improvement, I replaced the Dates list with a Map field. When I performed a count update for an object that existed before these changes, I noticed that the InfoMap field was seamlessly added. This flexibility makes it much easier to introduce new features (with null checks if we access the new info) and adapt to evolving processing requirements!

An old object which reflects a new field that was added to the schema

Conclusion

I plan to delve into the queries required for post-event reporting and discuss how to perform unit testing, as we’ll need to mock out the DB connection. I hope this journey has been a fun and enjoyable exploration of NoSQL! Find the full source code here (: