Implementing SAGA Pattern Using MuleSoft
We all work on multiple APIs, which collectively provide a business functionality. There are a lot of times when we work with numerous system APIs, which are interdependent. This is when the Data Consistency becomes very important. Let us take an example.
A user logs into an online shopping portal, to order for an item, that is priced at $3000. Now the user wants to avail an instant credit facility provided by the online portal. The user places an order, by utilizing the new instant credit facility and waits for the order. The Experience API sends requests to the down stream system api to update the Orders DB with the new order. Once this is completed, the experience api calls the credit check system API internally, to check the credit worthiness of the user ( using an external credit check portal) and then sends the request to the inventory service to check and update the inventory as per the Order details.
Now given the scenario that the credit check fails, the inventory is already updated with the reduction of the item from the inventory. However the orders api sees that the credit score is low hence the order is cancelled.
Now comes the real issue. The Orders DB and the Inventory DB are not in sync anymore as both are oblivious to each others state. To maintain the data consistency we are unable to to use the local ACID transactions as these two are in two different services. We are also not able to use the famous “Two Phase Commit” in summary where the commit of the first transaction depends on the completion of the second one.
This is one scenario where we need to implement SAGA pattern.
What is SAGA Pattern: The SAGA pattern is a design pattern used in distributed systems and microservices architecture to manage and coordinate long-running transactions or processes that involve multiple services. It provides a structured way to ensure data consistency and reliability in a distributed environment, where transactions may span multiple services or databases. The SAGA pattern offers a way to manage distributed transactions without relying on a traditional two-phase commit (2PC) protocol, which can be complex and has limitations in distributed systems. Instead, it embraces the idea of eventual consistency and uses compensation actions to handle failures and maintain data integrity.
SAGA pattern was first proposed in the year 1987. Here is the link for all those geeks who want the ORIGINAL version : https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf
SAGA pattern manages the transactions that tend to span across distributed services using a sequence of local transactions.
SAGA is implemented in two ways:
- SAGA as Choreography
- SAGA as Orchestration
Lets look at both of them in detail.
SAGA as Choreography:
In our example, the “Experience API” will be the controller of the SAGA
Here the Experience API first sends a request to the Orders API to update (Order Verification) and waits for the positive response. That is the sequence 1–3.
For the next part the Experience API sends a request to the Credit check API. This API internally call an external API to check the credit worthiness of the user and subsequently updates (temporarily) an internal Db to update a flag (Verifying) about the credit worthiness for future references. Then it sends back a response to the Experience API. This is the sequence 4–7.
For the third part the API sends a request to the Inventory API which updates the Inventory DB and sends back a response for the sequence 8–10.
Once the Experience API gets all the successful responses from the ????????
This is the way the entire SAGA is handled by one service for the positive scenarios.
Coming to the Negative Scenarios.
- Scenario One
Assuming that there was an error in the Credit Check of the User. In such cases the Credit Check API first updates the DB with the flag as (Verifying) and sends a negative response to the Experience API.
Here the Experience API does not send the third request to the Inventory API. It directly sends out two more calls to the Order and Credit APIs to update the DB with statuses as (Order Cancelled) and (Failed) respectively.
It also sends out a response to the user notifying about the status of the order.
2. Scenario Two
Assuming that there was insufficient items in the inventory. In such cases the Inventory API does not update the Inventory DB. It also sends out an error response to the Experience API with the error.
The Experience API sends out two more calls to the Order and Credit APIs to update the DB with statuses as (Order Cancelled) and (Failed) respectively.
It also sends out a response to the user notifying about the status of the order.
SAGA as Orchestration:
The SAGA Orchestration does not have a controller to direct and execute the transactions. Here each service is responsible for creating a separate event. It is the SAGA Execution Controller that keeps track of the events using a SEC log and executes roll backs in the events of failures.
The same workflow that was explained earlier, would look like this with the Choreography (Orchestration?) approach.
Now what if the Credit Check Fails?
In this case, the credit check service would inform the SEC about the failure. The SEC would now start the corresponding rollbacks and end the ordering process by setting it to fail.
Now from the second approach we see that implementing SAGA Choreography is much more complex as compared to implementing SAGA Orchestrations.
Finally before one decides to implement SAGA pattern, one should remember that all the complexities of each implementation needs to be weighed in before you choose. Also it is always easier to maintain and debug simple frameworks compared to the complex ones.
The choice between SAGA orchestration and choreography depends on the specific requirements and constraints of the system. Each approach has its strengths and weaknesses, and the decision often revolves around factors like system complexity, fault tolerance, and performance.
In summary, the SAGA pattern is a valuable tool for managing distributed transactions in microservices and distributed systems, offering a structured approach to ensure data consistency and reliability while accommodating the dynamic and decentralized nature of modern architectures.
