Event-Driven Orchestration using Camunda BPM and Kafka
Microservices architecture has become ubiquitous today for implementing enterprise applications because of many benefits like greater separation of concern, faster time to market, higher scalability etc. However as the number of microservices increases, organisations are faces with challenges :
- Increased complexity of integrating distributed services when business process span across multiple services.
- Difficulty in managing transactions and rollbacks across services.
- Lack of visibility needed for end-to-end governance of business process across different services.
If not handled properly, this will result in Death Star Architecture with tightly coupled services, increased complexity and governance nightmare.In this article we will take a sample “shopping cart” business process and explore solution options to implement this fictitious process by integrating different microservices — Location, Payment, Inventory and Order services.
Option 1: Event-Driven Choreography
In this approach business process logic is decentralised and spread across multiple microservices using the principle of “Smart Endpoints and Dumb Pipes”. Each service is connected to an event stream and listens to specific Kafka topic; whenever an event is published to topic, corresponding service takes appropriate action. Thus each service is aware of its own responsibility in the overall business process.
For example, when user initiates checkout in application UI, a Checkout event is published to Kafka, Location Service validates delivery address and on success publishes Payment event. Next Payment Service consumes this event and reserves amount for payment. In similar fashion whole business process is realised based on events published on the event stream.
This approach provides loose coupling and high cohesion. However, decentralisation has its own drawback:
- Difficult to Manage: With business process spread across multiple microservices maintenance and changes in business process become very difficult to manage. Development teams requires understanding of overall process flow. Also a change in process impacts multiple services in the chain.
- Lacks observability : Tracing execution path and process state involves diving into multiple services and logs. This makes finding and troubleshooting failures difficult.
- Difficult to rollback: As business process is spread across multiple microservices, handling transaction becomes difficult.
Option 2: Centralised Orchestration
As opposed to Option 1, herein a central orchestration service (implemented using BPMN workflow and REST integration) contains the entire business workflow logic, and issues commands to worker services and awaits response. So instead of worker services themselves triggering actions on each other via events, the central orchestration service triggers actions on the worker microservices directly.
This approach provides following advantages over Option 1:
- Easy to maintain: Business process is modelled using standard BPMN model. This explicit definition of business process facilitates easy maintenance and upgrades. Also business can communicate process flow in a standard manner which fosters collaboration between teams.
- Better observability: Since the entire business process is defined in the orchestration layer, this approach provides great visibility into the progression of the flow. BPM process engines like Camunda keep track of all process instances, process state, audit and statistics information.
- Support for transaction and rollback: BPM process engines provides support for transaction and rollback.
However this approach results in tight coupling between the orchestration service and worker services because of point-to-point communication. Orchestrator needs to be aware of service deployment (like URL) and change to services impacts it.
Option 3: Event-Driven Orchestration
This combines best of earlier two options, centralised orchestration is provided by BPMN model while interaction with worker services happens via event stream. Instead of point-to-point communication, orchestrator publishes events for worker services thus ensuring loose coupling.
As shown in the diagram above, the centralised orchestrator (ShoppingCart service) can be implemented as SpringBoot microservice with embedded Camunda Process Engine. Service Tasks (like Validate Address, Reserve Payment) in BPMN model can use Java Delegates to publish events to Kafka topics.
One downside of this approach is additional complexity compared to standard orchestration (option 2) which results in additional effort to implement and maintain. In most cases, though, this overhead is offset by the ease of defining and modifying business workflows as well as the loose coupling that it provides.
BPMN model
An illustrative BPMN workflow for “Shopping Cart” is shown below.
An illustrative compensation BPMN workflow is shown below:
A sample implementation is provided in a separate article.
Conclusion
This article analyses service choreography and orchestration patterns of integrating microservices for implementing business process. Details pros and cons of these patterns, and then shows how event-driven orchestration draws best from both of them to provide a better solution.
