Kafka Streams with Serverless Kafka
Splitting notifications according to their types
In this story, I will share my experience of implementing a Kafka Streams Application with Upstash Serverless Kafka platform.
For details of Upstash Kafka you can have a look at the documentation https://docs.upstash.com/kafka
Kafka Streams is a client library for processing and analyzing data stored in Apache Kafka. Kafka Streams provides capabilities to process the Apache Kafka data as streams.
I implemented a simple Kafka Streams application with pure Java connecting to Upstash Serverless Kafka Platform. The application reads a notification topic and splits the notifications according to their notification types.
Here is the architecture of the application.
Implementation is available in Github.
Let’s run the application.
- Forming the Kafka Cluster in Upstash:
a. Login to https://console.upstash.com/kafka and create a cluster
b. Create topics in cluster
2. Implementation
a. Dependencies
We use kafka-streams and kafka-client dependencies for Kafka operations, jackson dependency for json serialization and logback for logging.
b. config file
You can add config properties for connecting to Kafka Cluster in Upstash.
You can get these values from Upstash Console in notificationCluster.
c. NotificationApp
This is an application with a main method.
i. It produces three Notification messages with SMS, EMAIL and PUSH_NOTIFICATION types and sends to Apache Kafka notification topic
ii. starts a Kafka Streams instance that consumes the Notification messages from notification topic and splits the messages according to their notification types and forwards to related topics.
In this part, we split the stream into different branches according to notification type of message with split() and branch() methods and materialize the stream to relevant topic.
We are using the Kafka Streams DSL that is a high level API. Kafka Streams DSL provides abstractions on creating the source stream from topic and transforming the stream and sinking the stream to other topics.
You define the topology of your Kafka Streams application and it just does the related operations after you start the Kafka Stream.
You can check the details of Kafka Streams DSL and its transformation functions in https://kafka.apache.org/27/documentation/streams/developer-guide/dsl-api.html
3. See the results in Upstash
Upstash gives us chance to monitor the messages in Upstash console.
When we select the Messages tab in topic page, we can see that notifications branched according to their notification types.
And that’s it. Thank you for reading. Please do not hesitate to send comments, ideas and make corrections.
