Helidon 2.0 with Kafka
Reactive birds flock together
With Reactive Messaging support coming with Helidon 2.0, Kafka connector is one of the first messaging connectors available for both flavors of Helidon MP and Helidon SE. Helidon MP follows the MicroProfile specification to the letter and leverages all of the great features of CDI. With Helidon SE you can enjoy simplistic API, explicit configuration, and the “ no-magic” approach users enjoy about SE.
Configuration
Connecting Kafka usually requires a portion of non-trivial configuration. Orchestration of such configuration can be challenging. To make things easier, both flavors of Helidon support the same notation of external configuration for Reactive Messaging as defined by MicroProfile Reactive Messaging specification.
Let’s look at a simple example:
In this example we show a simple configuration of one incoming channel from-kafka, and one outgoing channel to-kafka both connected to the same topic. Let’s try to use the very same configuration with both Helidon SE and MP.
Kafka in Helidon MP
Helidon MP implements MicroProfile Reactive Messaging Specification so all you need to do is add the required dependency and annotate methods of application scoped bean.
While listening for messages is simple:
For sending messages you need to take in to account possible backpressure. Let’s use SubmissionPublisher with built-in buffer for such a task:
Kafka in Helidon SE
There is a dedicated API for using Reactive Messaging without a CDI in Helidon SE. While the API is not part of the Microprofile specification, it is compatible with connectors, configuration, message wrapping and acknowledgement. All that for transition between Helidon flavors to be as smooth as possible.
Listening for messages is as straightforward as in MP flavor:
We don’t need to bother with SubmissionPublisher for sending messages as SE Messaging API provides Emitter:
Acknowledgement
Sometimes it is practical to defer offset commit to the point where it is certain the message has been properly received. A lot can happen during async processing, while we can’t block a reactive stream, there is a need for other kinds of feedback than backpressure. It can be quite convenient to leverage the acknowledgement feature of the reactive messaging. Kafka connector is committing only offset of acknowledged messages when auto.commit is off.
Turn off auto-commit:
In this example we can see deferred acknowledgement causing delayed offset commit:
As you can see in this example backpressure is applied by requesting one message after the other so utilization is not exactly effective. In case of catastrophic failure during async consuming the non-consumed message is NOT acknowledged, so Kafka connector didn’t commit offset. Nothing is lost and messages can be consumed again after rebalance.
All that packed in 20 lines of readable functional code, you can admit that there is a lot of the config hidden, so what about that no-magic approach?
Ok, let’s be even more explicit.
Explicit Configuration
Helidon SE Reactive Messaging supports direct supplying of Helidon Config to either Publisher or Subscriber connector of any channel. You can see in the example below, that Kafka connector has its own Config builder. So you don’t have to look up most frequent client properties every time you are connecting to Kafka.
It’s all there in one snippet, Abracadabra!✨ no-magic it is!
For more info about Reactive Messaging in Helidon:
