Integrating Kafka with Flowable

Hi all. Not long ago, I wrote an article on Flowable — Overcoming Common Hurdles in Flowable. Since then, I have been using Flowable often.

In my current project, we are using an event-driven architecture orchestrated by Apache Kafka. In the scope of the project, I realised that Flowable would be quite useful.

But the pertinent question that I had in my mind was… can I use Flowable in an event-driven context? My concern was eased when I found out about the latest 6.5.0 release of Flowable. This new version introduced a new Event Registry engine which provided functionality to use Flowable in event-driven architectures and had out-of-the-box support for JMS, Apache Kafka, and RabbitMQ.

In this blog, I will try to summarise the steps for writing a Spring Boot microservice that starts a Flowable workflow when a message reaches a Kafka topic. If you are just starting out on your Flowable journey, check out my earlier article first. Otherwise, I will assume you have familiarity with Spring Boot, Apache Kafka, and Flowable - the three components that we will try to stitch together.

Let’s begin with the basics first. Here is the flow that I want to execute designed in Flowable modeller.

The service task executes when the “Start Event Registry Event” gets a signal from Kafka. If you are wondering what a “Start Event Registry Event” is, it's a new start event inside the modeller.

For the “Start Event Registry Event”, the two most important properties are the event key and the mapping from event payload:

Flowable supports JSON and XML as deserialisers when it reads the message from Kafka. The following is the expected JSON message in the Kafka topic.

{

“vehicleNumber”: “123”,

“carpark”: “3A”

}

From event payload, I have mapped it as:

The event property name field is the JSON field that you want to map. In my case, it’s “carpark,” which is of data type string. The variable name field is the variable that I want to store the value of the JSON field in Flowable. In my case, the variable name is also “carpark”. These are the variables that I can access and modify from delegateExecuton in the Java Delegate class by calling delegateExecution.getVariable(“carpark”) method. Now we are ready to export the model as a BPMN XML.

Now let’s move on to the Spring Boot application. I won’t go into the details of writing the Spring Boot application, but I will highlight the critical parts.

The first thing to do is to bring in the dependencies. Do note that I am using Spring Boot 2.2.6.RELEASE.

Here are snippets of the essential dependencies:

Ensure that the version is 6.5.0.

We will create a processes folder under resources in your Spring Boot application and put the BPMN XML there. The Flowable spring boot starter process looks for the XML in this location.

Next, create another folder under resources and name it “eventregistry”. This folder contains two kinds of files — an event file and a channel file.

What is an event file?

An event file is where you keep the event definition. The extension of the file is “event” and it looks like:

The key is an identifier for an event and matches with the key that we have already covered for “Start Event Registry Event”. Since this event is listening to a Kafka topic, it needs to have an inbound channel associated with it. The “inboundChannelKeys” is a list of keys which are identifiers for the inbound channel. The payload attribute holds the definition of the payload, the name, and the data type of the JSON fields. The “correlationParameters” should always be empty for a start event. Boundary events use this frequently and are generally a JSON field from the payload. When multiple process instances of this example process definition are running, how can an incoming boundaryEvent message match with one of the running process instances? That is what is defined by the correlation parameters.

What is a channel file?

A channel file is where you keep the channel definition. The extension of the file is “channel” and it looks like:

The key is the identifier of the channel and matches with the one mentioned inside “inboundChannelKeys” attribute in the event file. The channel type can be inbound or outbound. The topics attribute is a list of topics that the event should listen to, and “channelEventKeyDetection” is an identifier of the event that this channel is bound. The value matches the identifier in our event file. The event identifier can be a fixed value like what I used but can also be a JSON field from the payload.

Once we have placed the event and channel files under eventregistry folder, we have one more thing to do: we need to put the following properties in the pom.xml.

And that’s it… we are now ready! Run your spring boot application and you can start producing messages in the topic that you want the Flowable start event to listen. For my case, the topic name is “CARPARK_ENTER”. You can then see the Flowable process being created and the service task getting executed.

I hope this article helps you guys set up and use Flowable in your event-driven projects. For more developer resources on enhancing your workflow, check out PALO IT’s publications and feel free to reach out to me on social.

Till we meet again… happy coding!