What’s new in Azkarra Streams 0.8 The micro-framework that takes your Kafka Streams apps to the next level!
I am pleased to announce the release of Azkarra Streams 0.8. Azkarra has benefited from several internal optimizations since the last version. This new release includes some new features as well as some minor breaking changes of the public API.
What is Azkarra Streams ?
For readers that discover the framework: The Azkarra Streams project is dedicated to making the development of cloud-native streaming microservices based on Kafka Streams simple and fast!
I recommend you to check out these blog posts to discover all the many possibilities Azkarra has to offer:
With this new release, the Azkarra framework continues to take Kafka Streams to the next level by providing features designed to reduce the time-to-market of your streaming micro-services.
This blog post summarizes the most important improvements.
Kafka Streams and Server-Sent Events (SSE)
Most of the time, Kafka Streams is used to aggregate and join records from one or more topics. Also, Kafka Streams maintains internal states for stateful operations and states updates are sent to an output topic for further use.
But it can sometimes be useful to send updates directly to the target system, instead of having to go through Kafka (again). For example, this can be to notify or to trigger the update of a data-visualization dashboard. In such a use, we would simply want to push the data from our topology to the dashboard so that it can refresh in real time.
Server-Sent Events is precisely a technology that allows a client to receive a stream of events from a server over one persistent HTTP connection. SSE is a lightweight alternative to Websocket that provides an efficient unidirectional communication protocol with standard mechanisms for handling errors.
Azkarra v0.8 offers a new API to easily capture records from your KafkaStreams instance and send them over HTTP using SSE.
Now, let us take a closer look at this new API through this simple example:
(1) First, we have to either implement the EventStreamProvider interface or simply extend the EventStreamSupportclass like in the example.
(2) Next we need to create and register an EventStream which is used to queue records to be sent. In the example, we register an EventStreamwith name word-count. Also, the EventStream is configured to drop records when the queue size is reached .
(3) Then, we use the foreach method to send records to the EventStream.
(4) Finally, to start receiving records you can use the new REST API : /api/v1/streams/:applicationId/subscribe/:eventStream.
And it’s as simple as that!
Waiting for Brokers to be Ready
Azkarra Streams 0.8 introduces a new StreamsLifecycleInterceptor called KafkaBrokerReadyInterceptor that wait for some brokers to be ready before starting your KafkaStreams application.
This new interceptor is easily enabled with the property kafka.ready.interceptor.enable=true.
In addition, this interceptor can be configured with the following properties :
kafka.ready.interceptor.timeout.ms: Wait until brokers are available or this timeout is reached (default is60000).kafka.ready.interceptor.retry.backoff.ms: The amount of time to wait before verifying that brokers are available (default is1000).kafka.ready.interceptor.min.available.broker: The minimum number of brokers that must be alive for the interceptor to stop waiting (default value is1).
Exporting State Store offset lags
Since Kafka Streams 2.5, the public API exposes the new method KafkaStreams#allLocalStorePartitionLags to get the lags for all store partitions (active or standby) local to the KafkaStreams instance. Developers can use this method to make decisions on whether a local state store which is not update-to-date (i.e: data is considered to be stale) is fresh enough for querying.
You can now expose the values returned from this method directly into Kafka for monitoring purposes using the provided MonitoringStreamsInterceptor.
This interceptor was introduced in Azkarra 0.7 and allow to periodically publishes state of the local streams instances into Kafka in the form of events that adhere to the CloudEvents specification.
Because it’s not recommended to invoke this method too often, by default store-partition-lags info is not exported. To enable you should set the following property:
monitoring.streams.interceptor.info.enabled.stores.lag=true
Here is an example of an event published by Azkarra :
Azkarra and Kotlin
Kotlin is being used by more and more projects. Good news you can also choose Kotlin to develop Azkarra-based applications!
Indeed, Azkarra had introduced an improvement in a previous version that had broken the compatibility with Kotlin language. This issue has been fixed in Azkarra 0.8. Moreover, a new Maven Archetype has been added to create a simple Azkarra project structure for Kotlin.
$ mvn archetype:generate -DarchetypeGroupId=io.streamthoughts \
-DarchetypeArtifactId=azkarra-quickstart-kotlin \
-DarchetypeVersion=0.8.0 \
-DgroupId=azkarra.streams \
-DartifactId=azkarra-project \
-Dversion=1.0
-Dpackage=azkarra \
-DinteractiveMode=falseOverview of other new features
WaitForSourceTopicsInterceptor
The WaitForSourceTopicsInterceptor now supports the two following properties :
wait.for.topics.timeout.ms: Wait until all topics are created or this timeout is reached (default isLONG.MAX_VALUE).wait.for.topics.exclude.patterns: The list of topics (regex) for which the interceptor should not wait.
Allow to only return successful records from Interactive Query
Azkarra provides built-in REST API for querying state stores. The key-value records returned from the REST API are wrapped into a JSON payload that provides information about the query results (e.g.: number of total records, time took, etc);
Azkarra now provides the following REST API that you can use to retrieve only the KV records :
/api/v1/applications/:streams/stores/:store/recordsAdd new ConfValue annotation
Azkarra Streams 0.8 introduces the new annotation ConfValue which can be used to annotate a component implementing the Configurableinterface for defining the default config properties that will pass to the method configure.
Support for Apache Kafka 2.6
Finally, Azkarra Streams is always built on the most recent version of Kafka. Therefore, this new release adds support for version 2.6.
Join the Azkarra Streams community on Slack
The Azkarra project has a dedicated Slack to help and discuss the community. Join Us!
Conclusion
I would like to thank you, the early adopters of Azkarra, who through their feedback and support help the project to become more and more robust after each new version.
Please, share this article if you like this project. You can even add a ⭐ to the GitHub repository to support us.
Also, the project is open for contributions. So feel free to propose your ideas or project needs and of course to create pull requests (PR).
Thank you very much
About Us :
StreamThoughts is a French IT consulting company specialized in event streaming technologies and data engineering, which was founded in 2020 by a group of technical experts. Our mission is to help our customers to make values out of their data as real-time event streams through our expertise, solutions and partners.
We deliver high-quality professional services and training, in France, in data engineering, event streams technologies and the Apache Kafka ecosystem and Confluent.Inc Streaming platform.
