Camunda external tasks — a powerful tool for creating applications with a fault-tolerant and scalable architecture
In Tinkoff we use the Camunda + Spring framework for the development of business process automation systems. We visualize them in the form of flowcharts by BPMN (Business Process Management Notation).
The most frequently used element on our schemes is service tasks (rectangle with a gear). Camunda supports two ways to perform service tasks:
- With the help of a synchronous Java code.
- With the creation of external task.
The second way allows you to perform tasks with the help of external systems. For example, if you need to call one camunda application from another or even delegate work to some external system.
This is useful when you are going to reuse logic in multiple applications. Or when you want to stick to the microservice architecture. For example, by separating services that do business processes from services that perform technical tasks such as report generation or mailing out.
Along with this capability, external task provides a scalable, fault-tolerant architecture. To understand what this is all about, you first need to understand how the external task works at the BPMN and application levels.
External task in BPMN
External task involves creating a task that can be performed by an external handler. The essence of the external task pattern is that:
- The process that “orders” the task execution simply adds its “order” to the database.
- Some abstract handler asks Camunda for the task to be processed, at the same time assigning the task to himself so that another handler could not execute it.
- After execution of the task the handler tells camunda the result of execution (successful or unsuccessful).
In the diagram above, I described a fictitious process where we want to get a list of users, send them ads and count the number of requests after 2 hours of marketing. And if there are more than 10 requests, increase the sample for the next mailing list.
I want my application on camunda to be responsible only for business processes, and any other application will be responsible for email distribution. In that case, the external task pattern fits me perfectly. In my process, I’ll just create an email task and wait for some external handler to perform it.
In order to create an external task in the schema, it is necessary:
- Create a regular task.
- Change its type to a service task.
- Set implementation to external.
- Specify the value of the Topic field.
Topic is the name of the queue, in which tasks of the same type will be added and to which the external handler will subscribe.
Now that there is an external task in the process, you can run it, but it will not run because no one is processing it.
External tasks worker
The external task pattern is good because it allows you to implement the processing of tasks in any language, with any tools that can perform HTTP requests.
Below is an example from the camunda blog. The example implements an external Javascript handler that asks camunda every 20 seconds for a list of tasks to be processed. If there are tasks, it mails them and notifies camunda when the task is finished.
As you see from the code above, fetchAndLock and complete are the key methods for handling external tasks. The first method requests a list of tasks and commits their execution to itself, while the second method informs you when the task is finished. There are other methods besides these two, you can read about them in the documentation.
Camunda external task client
To implement external tasks processing, camunda provided clients in Javascript and Java, which allow you to create external task handlers literally in a few lines. There is also a detailed guide that describes the basic principles of external task processing, again with examples in Javascript and Java.
An example of an external handler implementation using the ExternalTaskClient:
If your task requires you to run an entire process rather than just perform some synchronous action, you may do it, for example, by running the process through RuntimeService:
In this example, the external tasks handler (EmailWorker) runs the SendEmailProcess process when a task is received.
Let’s assume that this process performs some actions necessary to send a mailing list and finally calls EmailResultDelegate, which in turn completes the external task.
Architectural advantages of an external task
It should be noted that there is a way to start the process in another camunda application in a simpler way: POST: /rest/process-definition/key/${id}/start.
When you use REST, you have no transactional guarantees. But with an external task we also work with REST, what is the difference then?
The difference is that we don’t call the external service directly, we just publish tasks that can be processed. Let’s look at an example:
Some external handler picks up a task that is now assigned to him, but when a response is received, the connection is dropped. The camunda side now has a blocked task that will not be processed because the external handler has not received a response. But that’s okay: in camunda, for external tasks, there is a timeout at which the task will return to the queue again and someone else will be able to process it.
Now let’s look at the case when the external handler received the task, executed it, and called the complete method, which ended up with an error due to network problems. You will now be unable to figure out whether or not the task was successfully completed in camunda. You can try again, but there is a chance that network problems will continue.
In that case, the best solution is to ignore the problem. If the task was successful after all, then everything is fine. If not, then at the end of the timeout the task will be available for processing again. But this means that your external handler must be independent or contain logic for deduplication of the task.
A similar problem may occur when starting a new process, so you should check your existing instances with the same data, such as businessKey.
Besides high fault tolerance, external tasks allow you to easily scale external handlers and implement them in any programming language. In this case, the pattern helps to implement microservices so that they affect each other as little as possible, thus increasing their stability.
You can read more about the external task here:
