Distributed Transaction Management in Microservices World

Modernization is one of the key initiatives these days for any large organization. Monolithic to Microservice Architecture transformation is the backbone of such modernizations.

A monolithic application is built as a single unit which has it’s own disadvantages to naming a few, the application is too large and complex to fully understand and made changes fast and correctly, redeployment of the entire application on each update, Continuous Integration, and Continuous Deployment is difficult, Monolithic applications have a barrier to adopting new technologies but it has advantages too such as starting a new project and developing it is simpler using monolithic architecture, components like frameworks, templates, or scripts can be easily applied and out of them, one of the most impactful capability is Transaction Management.

Microservices architecture has various advantages but distributed transaction management is a challenge and it is recommended to avoid distributed transaction management across microservices but if we deal with large systems then in order to keep data consistencies and ACID properties, transaction management is the key.

Microservices architecture vs Monolithic architecture

Before we jump to the Distributed Transaction Management in the microservices world, let’s understand what the major difference between microservices and monolithic architecture is, and how the transaction management need originated.

Web Application development is an old (the 1970s) concept that started with digitalization to solve various industry problems using software applications. Earlier we used to have desktop-based applications and then the trend started to migrate them as web-based applications using a client-server technology. As technology advanced and more process automation was required, most of the companies continued enhancing existing applications which resulted in “fat” monolithic enterprise applications. A big thanks to microservices architecture which initiated critical thought processing to break applications into APIs and easily manageable components.

Enterprise Applications are built in three parts: a database (consisting of many tables usually in a relational database management system), a client-side user interface (consisting of HTML pages and/or JavaScript running in a browser), and a server-side application. This server-side application will handle HTTP requests, execute some domain-specific logic, retrieve and update data from the database, and populate the HTML views to be sent to the browser. It is a monolith — a single logical executable.

The advantages of three-tier applications in creating scalable and robust applications are made feasible by transaction processing systems. The ability to distribute the components that make up applications amongst separate servers without explicitly having to develop for that architecture is another advantage of transaction server processing. This alleviates the developer from having to support these characteristics explicitly.

When a transaction processing system creates a transaction, it will ensure that the transaction will have certain characteristics. The developers of the components that comprise the transaction are assured that these characteristics are in place. They do not need to manage these characteristics themselves. These characteristics are known as ACID properties. ACID is an acronym for atomicity, consistency, isolation, and durability.

Now, let’s look into Microservices architecture, it is an architectural style that structures or breaks a monolithic application as a collection of loosely coupled services to implement business capabilities. The Microservice architecture also speeds up continuous delivery/deployment of large, complex applications. In simple terms, start thinking about breaking your application into independent components to serve business capabilities at a fine-grain level.

Microservices architecture has various advantages but distributed Transaction Management is a challenge, and it is suggested to avoid transactions across microservices but if we deal with large systems then in order to keep data consistencies efficient transaction management is a crucial process.

What if Transaction Management is not handled efficiently in Microservices architecture?

In order to understand the impact of Transaction Management, let’s take a simple scenario where we have 3 distinct microservices with its own independent business capability which connects to different databases. Our business requirement is, we need to ensure all 3 steps are completed and if any one of the required steps is failed then we would like to rollback all 3 steps. Out of these 3 steps, 3rd step is not critical but for eventual consistency, we need to make sure it is completed.

In order to understand the challenges let’s take the example of the Venue Booking System using Microservices, there could be multiple failure points that can impact the ACID properties of the Venue Booking System.

1. What if any of the required steps are not complete, how should it be accomplished in Microservices?
2. What if, one is booked twice for the same date & time?
3. What if, payment is deducted without booking a venue?
4. What if the booking is completed but the user is not notified?
5. There could be other similar system issues…

All these issues can be handled easily in the monolithic application (if we are using the 3 Tier Architecture Model) but in order to handle such scenarios in microservices architecture, we need to design a solution that can manage distributed transaction management efficiently.

Solution Approach for Distributed Transaction Management:

Distributed Transaction Management is a common problem in the industry and there are patterns existing to solve this problem, additionally, there could be some custom patterns. Following are the ways to handle transaction management between microservices:

Approach # 1

Avoiding distributed transactions across microservices — This is an ideal situation, but it may not be an ideal situation, especially if you are transforming from Monolithic architecture, so the following approaches can be considered.

Approach # 2

Manage transactions within Process APIs — This is a hybrid solution and is mainly applicable if organizations are in process of moving away from Monolithic and also, it is a good solution if are having complex processes that are very much inter-dependent. In these situations, developers can build Process APIs and orchestrate logical business functionality within the API. All complex transactions are managed within API and can eliminate needs for distributed transaction management and can be a better one from a performance perspective as we can avoid multiple hops between APIs, but this is not a true microservice architecture.

Approach # 3

2/3 Phase Commit Approach — This is one of the most important patterns in distributed transaction management using coordinator and it will consist of the following phases:

1. Prepare Phase: During this phase, all participants of the transaction prepare for commit and notify the coordinator that they are ready to complete the transaction
2. Commit Phase: During this phase, either a commit or a rollback command is issued by the transaction coordinator to all participants
3. Compensating Transaction (Rollback): During this phase, rollbacks are executed as a part of compensating transactions and eventual consistency.

2/3 Phase Commit is a very strong consistency protocol. First, the prepare and commit phases guarantee that the transaction is atomic. The transaction will end with either all microservices returning successfully or all microservices have nothing changed. Secondly, it allows for read-write isolation. This means the changes in a field are not visible until the coordinator commits the changes.

Following are challenges with this approach:

• Too many failure points, whole the solution is dependent on the coordinator, and if the coordinator is failed then everything is broken.
• If backend services are not responding, then the approach can be in-efficient.
• While 2/3 Phase Commit has solved the problem, it is not really recommended for many microservice-based systems because it is synchronous (blocking). The protocol will need to lock the object that will be changed before the transaction completes.
• Locking could become a system performance bottleneck. Also, it is possible to have two transactions mutually lock each other (deadlock).

Approach # 4

Eventual Consistency and Compensation / SAGA Design Pattern — In this pattern, the distributed transaction can be implemented using an asynchronous pattern, this is the most reliable and scalable solution, also it resolves some of the changes caused by the 2/3 Phase Commit Approach. In a Saga pattern, the distributed transaction is fulfilled by asynchronous local transactions on all related microservices. The microservices communicate with each other through an event bus.

This pattern has 2 approaches choreography and orchestration.

1. Event-based choreography, in this pattern each service produces and listens to other service events and decides if an action should be taken or not. All communications are initiated using the choreographer and responses are received by Reply Channel.

Events/Choreography is a normal way to implement Saga’s pattern. It does not require more effort. It is simple and easy to understand. In this, all participants are loosely coupled because they don’t have direct knowledge of each other.

However, this can become confusing if you add more steps in the transaction because it’ll become difficult to trace which services consume which events. In addition, it may create a cyclic dependency between services as they must subscribe to one another’s events.

1. Orchestration: In the orchestration approach, we define a new service with the sole responsibility of telling each participant what to do and when. The saga pattern orchestrator communicates with each service in a command/reply style telling them what operation should be performed.

Approach # 5

Phase commit approaches using the asynchronous pattern for failover logic — This is a custom pattern that takes care of transaction management using microservices and in case of any failure event, compensating the transaction can be executed as needed and asynchronous failover logic can be used leveraged for eventual consistency. This seems to be the most preferred method due to the availability of failover logic, but it has similar disadvantages as the 2 Phase Commit approach.

Conclusion:

In recent times, organizations are moving towards the cloud, and to support quick time to market, loosely coupled architectures are preferred. So, all new solutions are being designed and developed using microservices architecture, at the same time many organizations are transforming from monolithic to microservices architecture. Distributed Transaction Management is a crucial part of microservices architecture. Even though it’s recommended to avoid distribution transaction management but practically it’s near to impossible to avoid this.

Distributed Transaction Management is a complex process to manage transactions and achieve ACID properties due to challenges like locking, compensating transactions, and eventual consistency. Transaction management can be achieved using 2/3 Phase Commit, SAGA, Customer, and Hybrid design pattern mentioned, however selection of the right design pattern depends on the use case.

#microservices