Couchbase Transactions with Spring Boot 3.0

Original Image: link

In this article, I will try to explain how we implemented Couchbase transactions with @Transactional annotation in a Spring Boot application.

First, I will give some details and the reasons why we decided to use Couchase transactions in our systems. After that, we will look at how to implement transactional features in a Spring Boot 3.0 application.

Content
· The Problem
· Possible Solutions
  ∘ Compounding the Data
  ∘ Custom Management
  ∘ Couchbase Transaction
· Implementing Transactions In Couchbase
  ∘ Required Dependencies
  ∘ Couchbase Configuration
  ∘ Managing MDC Variables with Custom Couchbase Transaction Manager
  ∘ Document Object and Repository
  ∘ Business Service Logic
· Performance Results
· Conclusion

The Problem

As the Delivery Core team at Trendyol, our main domain object is shipment. For every order, we create and manage the lifecycle of shipment objects. We use Couchbase for storing and managing these objects.

Until recently, we could manage all data manipulation processes on shipment documents atomically, because one order could be represented with one shipment in our domain. Thanks to that we only needed to manage one shipment document at a time in our business services.

But a new business feature has come and changed our shipment management logic a bit. According to this feature, we have to create multiple shipments for an order, and also this creation process must be handled atomically to keep the database in a consistent state. So a new problem has arisen for us, how can we manage these shipments as a whole?

Demonstration of the problem

Possible Solutions

We worked on some possible solutions and compared the benefits and disadvantages. I won’t give every option which we tried, for keeping the article’s context clear I will explain only a few of them in this section. These are:

• Compounding the Data
• Custom Management
• Couchbase Transaction

Compounding the Data

Before using transactions in a NoSQL database, the first thing we could do is combine the data. For our use case, we tried to put shipments in an array to combine them as one document. Collecting whole data in one document gives us the ability to manage database operations atomically.

Compounding the Data

But after analyzing the impact of this change on our system, we realized that holding multiple shipments on the same document needs a huge development effort on our microservice architecture. We need to adapt all of them to manage the new structure. But we didn’t have much time to implement this feature. So we focused on the other options.

Custom Management

We thought that maybe we could implement a custom management system that supports atomicity for the shipment management processes. For the happy path, it looks very easy to create a logic like that. But when we started to think about error cases we realized that it is a complicated logic to implement. Some problems with this solution are these:

• We need to cover all failure cases and create compensating processes for some of them.
• It is too hard to manage isolation. After creating shipments, we produce Kafka events via connectors. In our case, we need to publish events when two shipments are inserted successfully. If we create shipments without isolation, there will be a problem for our system.
• Even if we write compensating process to roll back an operation, it is possible to face an error issue during the compensating process. So there is a risk of putting our database in an inconsistent state.

For these reasons, we give up this option.

Couchbase Transaction

After some research, we realized that Couchbase supports ACID transactions natively.

If you have ever used RDBMS before, you are probably familiar with transactions and ACID principles. For multiple data manipulation operations which must be completed entirely, we can use transactions. Transactions help us to commit or roll back a sequence of data manipulation operations as a whole. This type of atomic transaction is commonly referred to as an ACID transaction. ACID is an acronym for describing the basic properties of an atomic database transaction: atomicity, consistency, isolation, and durability.

Most of the RDBMSs have been supporting transactions for a long time, but how does Couchbase, as a highly scalable distributed document-oriented NoSQL database, manage transactions?

Couchbase has been supporting ACID transactions since version 6.5 and they continue to increase the performance of transactional operations on both the server and client side. Couchbase manages transactions efficiently, reliably, and securely. In this article, I won’t give the details about the architecture of the transaction management system Couchbase uses, but if you are curious about it, you can look at this great blog here.

With this solution, we don’t need to manage all failure cases and isolation issues because Couchbase natively supports ACID principles. Thanks to that we can save shipments or roll back them as a whole, as a result of this we can keep our database in a consistent state.

But of course, it comes with an overhead which we can accept for our use case. Transactional operations add some latency to our response time but as I said for our use it is acceptable because of the benefits.

And also with Spring Data and Couchbase SDK, it is very easy to manage these transactional operations. I will explain how to implement transactions using a declarative way with @Transactional annotation in a Spring Boot 3.0 application in the next step.

Implementing Transactions In Couchbase

There is enough information I guess :) let’s see how we implemented this feature…

Required Dependencies

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.0.5</version>
</parent>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-couchbase</artifactId>
    </dependency>
<dependencies>

For using Couchbase Transaction integrated with Spring’s @Transactional annotation, we need to use Spring Data Couchbase version 5 or above. In this example, we use spring-boot-starter-parent version 3.0.5, from that parent we can get spring-boot-starter-data-couchbase version 3.0.5, and spring-data-couchbase version 5.0.4 transitively.

With Spring Boot 3.0, all Spring codes have been updated to Java 17 version. So we also need to use Java 17 version at least in our project.

Couchbase Configuration

@Configuration
@RequiredArgsConstructor
@EnableTransactionManagement
@EnableCouchbaseAuditing(auditorAwareRef = "auditorAwareRef", dateTimeProviderRef = "dateTimeProviderRef")
public class CouchbaseConfiguration extends AbstractCouchbaseConfiguration {

    private final CustomCouchbaseProperties customCouchbaseProperties;

    // username, password, and connectionString getter methods are removed for keeping the code clean.

    @Override
    public void configureEnvironment(ClusterEnvironment.Builder builder) {
        BucketProperties transactionMetadata = customCouchbaseProperties.getTransactionMetadata();
        TransactionKeyspace transactionKeyspace = TransactionKeyspace.create(
                transactionMetadata.getBucketName(), transactionMetadata.getScopeName(), transactionMetadata.getCollectionName()
        );
        builder.transactionsConfig(TransactionsConfig.metadataCollection(transactionKeyspace));
    }

    @Bean(BeanNames.COUCHBASE_TRANSACTION_MANAGER)
    public CouchbaseCallbackTransactionManager couchbaseTransactionManager(CouchbaseClientFactory clientFactory) {
        return new DeliveryCouchbaseTransactionManager(clientFactory);
    }

    @Bean
    public NaiveAuditorAware auditorAwareRef() {
        return new NaiveAuditorAware();
    }

    @Bean
    public NaiveDateTimeProvider dateTimeProviderRef() {
        return new NaiveDateTimeProvider();
    }

}

This configuration part is crucial to integrate Couchbase into our Spring Boot project. Let’s explain the details of this config:

• @EnableTransactionManagement: Enables Spring’s annotation-driven transaction management capability.
• @EnableCouchbaseAuditing: This annotation let us use Spring Auditing annotations like @CreatedDate, @CreatedBy, etc. with Couchbase domain objects. The default values of auditorAwareRef and dateTimeProviderRef parameters are empty, in this example, we define our custom auditors so we also need to pass bean names into these parameters.
• configureEnvironment(): In this method, we define which bucket will Couchbase use for storing transaction metadata information. For transactional operations, Couchbase holds binary metadata info for each v-bucket. And for every transactional operation, this binary data is updated. By default, these binary data are held in the default bucket, but if you don’t want to mix your domain data with these binary data, you can separate them. In this method, we give our transaction metadata info with bucket, scope, and collection name to the cluster environment builder. Thanks to that all binary data used for managing transactions will be held in a different bucket.
• couchbaseTransactionManager(): This method is actually not required. If you enable the transaction, CouchbaseTransactionManager will be automatically enabled. But for our use case, we needed to update the native nature of the default transaction manager, so we defined our custom transaction manager. I will explain why we did this in the next part.

Managing MDC Variables with Custom Couchbase Transaction Manager

If you enabled transactions with @EnableTransactionManagement, CouchbaseCallbackTransactionManager is automatically initiated as a bean by default. But why, the default behavior was not enough for us?

The reason is the logging system that we use in our projects. Logs are very important to investigate problems. We use log4j2 as a logging framework, and for searching logs that are related to each other we label them with MDC.

When you put a variable into MDC this variable only exists in that local thread, if you create a new thread in another thread, MDC variables will not be moved automatically.

When we mark our methods with @Transactional annotation, spring creates a proxy class, and manages transaction features in that proxy class. When we call a transactional method from another bean, spring actually calls its proxy class’s method under the hood. In the end, this proxy method calls the actual method in another local thread.

Proxy Call

So we realized that when we call a Transactional method we lose our MDC variables and cannot tag our logs anymore. So we come up with a solution as below.

public class DeliveryCouchbaseTransactionManager extends CouchbaseCallbackTransactionManager {

    public DeliveryCouchbaseTransactionManager(CouchbaseClientFactory couchbaseClientFactory) {
        super(couchbaseClientFactory);
    }

    public DeliveryCouchbaseTransactionManager(CouchbaseClientFactory couchbaseClientFactory, TransactionOptions options) {
        super(couchbaseClientFactory, options);
    }

    @Override
    public <T> T execute(TransactionDefinition definition, TransactionCallback<T> callback) throws TransactionException {
        Map<String, String> contextMap = MDC.getCopyOfContextMap();
        return super.execute(definition, status -> {
            MDC.setContextMap(contextMap);
            T result = callback.doInTransaction(status);

            if (status != null) {
              Optional.ofNullable(MDC.getCopyOfContextMap())
                      .ifPresent(cm -> cm.keySet().forEach(MDC::remove));
            }
            return result;
        });
    }
}

When a transactional method is called, execute() method in DeliveryCouchbaseTransactionManager is also called. We override this method to copy all MDC variables from the parent thread to the new local thread. And at the end of the method, we also clean these variables to avoid MDC variable pollution. Thanks to that solution we could copy all MDC variables into the methods marked with @Transactional.

Document Object and Repository

These code examples are added for the demonstration of object details used in the Business Service Logic section.

@Data
@Document
public class ShipmentDao {
    @Id
    private Long id;

    private Status status;

    private String shipmentNumber;

    private String orderNumber;
    
    private Long cargoId;

    @CreatedDate
    private ZonedDateTime creationDate;

    @LastModifiedDate
    private ZonedDateTime lastModifiedDate;

    @Version
    private Long version;
}

@Repository
public interface ShipmentRepository extends CouchbaseRepository<ShipmentDao, Long> {
}

Business Service Logic

Well, we defined all required configurations. Now we can create our service logic which uses @Transactional annotation.

@Service
@RequiredArgsConstructor
public class ShipmentCreateService {

    private final ShipmentRepository repository;

    @Transactional
    public void createShipments(List<CreateShipmentRequest> requests) {
        List<ShipmentDao> shipmentDaos = requests.stream().map(this::generateShipmentDao).toList();
        repository.saveAll(shipmentDaos);
    }

    private ShipmentDao generateShipmentDao(CreateShipmentRequest request) {
        return ShipmentDao.builder().shipmentNumber(request.getShipmentNumber())
                        .cargoId(request.getCargoId())
                                .statusId(request.getStatusId()).build();
    }
}

As you see above, we defined thecreateShipments() method which is marked with @Transactional. When we configure Couchbase correctly, Spring Data Couchbase manages transactions for all methods marked with @Transactional annotation. Thanks to this, if one shipment creation gets an error, the whole creation process will be rolled back so we can avoid putting our database in an inconsistent state.

Important Note: When you throw an exception inside of the method marked with @Transactional annotation, Spring Data Couchbase automatically wraps it with TransactionSystemUnambiguousException. If you use any type of exception handler, you need to be aware of that.

Performance Results

Before we deploy this feature to production, first we did a load test to see the performance of transactional operations on Couchbase. By the way, the Couchbase database which we used during the load test was version 6.6. According to the results, transactions added overhead to the process. As you see below, the transactional feature added approximately 15 ms to the response time.

Non Transactional Results

Transactional Results

Conclusion

Using transactions with Couchbase is very easy, with Spring Data Couchbase 5. As you can see from the test results, the transaction added a significant amount of time to our total response time. For this reason, we only use @Transactional annotation on mission critic methods only, not for methods that require low latency, and high performance. In those methods which require low latency, we try to use single atomic operations for now.

According to this blog, the Couchbase team introduced fast transactional support on the server side with version 7.0. We are planning to upgrade our Couchbase servers to version 7.0. After upgrading the server versions, we will retry this load test and, I’ll also share the new results with you in this article.

Special thanks to Kaan Taş and Hakan Eröztekin for helping me to write this article.

Thank you for reading.