Batch Processing with Camunda
During the implementation of Enterprise Systems the requirements of batch processing needs to be met regularly. This article discusses possibilities of implementation Batch Processing using Camunda process engine.
Camunda BPM Engine is a developer-friendly, modern, light-weight BPMN 2.x process engine written in Java. It allows for efficient and effective implementation of the orchestration layer in business process automation systems by embedding the process engine into the application. In doing so, it provides not only the execution engine for BPMN 2.x, but a couple of very useful facilities and components which can be used in the resulting system.
Image, you are implementing a system which need some kind of notification capabilities on the execution of certain steps in your business process. For example, you may wish to send text messages or emails based on some conditions. In order to do so, the recipient address, the subject and the message body needs to be passed to the corresponding service implementation which is responsible for the transmission of the message. This transmission service is out of scope of this discussion, but as every communication component, there can be errors during transmission.
In the following sections, three approaches are discussed and some code examples are provided. In doing so, I focus on a SpringBoot stack, but similar can be done in JEE or other compatible frameworks.
The Java way
Batch processing is an old topic well implemented in most ecosystems. Java provides multiple frameworks for implementation of the aspects of batch processing. First of all, the execution of some functionality asynchronous to the main program execution is provided by the Java Concurrency framework, which is a part of the Java SDK. This is very low-level, but sometimes already sufficient to implement the required use case. In enterprise systems frameworks like JEE or Spring are used which both provides solution for running jobs in a background.
In SpringBoot, you can utilize the spring-boot-starter-batch, create an item processor and configure a job for execution of tasks. In our example, the item processor will call the transmission service and pass the required information to it.
The disadvantage of this approach is that you have to take care of the jobs if they fail and implement the failure-handling and recovery yourself. The same argument holds for the operation and management of jobs, their statuses and retries. Those can be pretty time-consuming and error-prone to implement.
For more details, how to implement a batch service in Spring Boot see https://spring.io/guides/gs/batch-processing/
Another approach is to integrate a batch processing framework, like Apache Camel. The resulting solution might get much more powerful and comprehensive, but this is a much bigger effort then. For details on that please refer to https://www.baeldung.com/apache-camel-spring-boot
The Camunda way
The BPMN way
One of the naive things to do is to use the core capability of the process engine and create a BPMN process model for the execution of the batch. This process is then used as a “technical” support-process which can be called from any business process in the engine. It can be started by a message or by a call-activity, depending on the use case.
One could argue that encapsulating a simple service call into a business process just for making it executable asynchronous and fault-tolerant is to much of effort. From the operations and management point of view, the implementation as a business process advantages from the uniformity of error-handling and other standard operations which can be applied to a business process.
In our example, the process contains a service task to render the message and send notifications (see example implementation of such a process below). If the notifying process is interested in the status of the execution of the notification, a call activity can be used.
In addition, more business requirements can be implemented in the notification process, for example, delayed notification or notification retry during a certain period of time.
The main advantage of this approach is that the batch process becomes a first-class citizen in the Camunda process engine. The process has a persistent instance, encapsulating the execution and state. It can be accessed via Camunda API or interact with other running process instances. In addition, this means that it can be handled by the standard tooling of the business process engine and is visible in the operations tool Camunda Cockpit.
The Job Executor way
Another approach of implementing batch tasks based on Camunda process engine is utilization of the Camunda Job Executor. This facility is a part of Camunda BPM engine and is used internally by the engine every time, some tasks needs to be executed asynchronously. The job executor can be tuned and configured using standard Camunda configuration mechanism in order setup many parameters to fulfill your requirements. In doing so, the custom jobs get executed along with Camunda jobs. Its execution is available using the Camunda API.
Apparently, the creation of a custom job is not a part of official Camunda API, but still can be executed from the application code. The implementation itself is a subject of a separate post.
Summary
Batch processing functionality in a project using Camunda process engine can be implemented in several ways.
A pure Java approach seems to be very low-level and error-prone and lacks management and operation functionality. Try to avoid this… Introduction of a specific Batch Processing framework is advisable, but adds complexity to the overall solution.
Camunda provides two alternatives to those approaches. Implementation of a support process allows for handling of Batch Processing steps as first-class citizen with all full-featured functionality in management and operations. Even if a little costly from the implementation point of view, it may be worth to invest in it, especially if some business logic regarding the execution or failure compensation needs to be implemented.
Another powerful tool for implementation of Batch Processing is Camunda’s Job Executor. Even not accessible over the public API of Camunda a job can be easily created as described in this article. The downside of direct programmatic Job creation is that the management and operation steps are available over the Management and Runtime Java and REST API only. By default, there is no support for them in Camunda Cockpit, as it is the case if implementing a support BPMN process, but it can be developed as a Cockpit plugin, if needed.
