Apache Kafka Competing Consumer Pub/Sub Pattern with Mule4 Part2

<SimplePubSubPattern><BroadcastingPattern>

As we are aware of Simple Pub/Sub pattern Let's explore Competing Consumer pattern in Kafka.

In Consumer Competing pattern, once producer publishes message to topic, anyone of the consumer will consume the message, order of consumption is not guaranteed.

Why Do we need Competing Consumer pub/sub pattern?

For example, if we have a requirement that message should be processed only once by any one of the consumers, then we can choose this pattern.

Uses Case for Demo (Video) Competing Consumer Pattern

Competing consumer pub/sub pattern

Here Producer produces message, but either ConsumerA or ConsumerB will consume the message from topic.

Demo To create Free Kafka with Confluent Cloud

please follow the steps in Video if you don’t have Kafka in handy for demo, where you can create Kafka topic with Confluent. Based on the link I have created Kafka cloud account for bootstrap server, username and password for my publish and consume connectors.

Anypoint Studio Flow

Studio Flow for competing Consumers

point to remember in while achieving the above pattern, both Message listeners should have same Consumer Configuration (i,e. same groupId and Topic)

(1) Configure message to publish to kafka

message to publish

(2) Configuration to Publish Message to Kafka

topic Name: competing

Publish Kafka Configuration

Consumer-A flow

(3a) Once Message is published Successfully, we can Consume the Message as below

groupId : ConsumerAB

topic: competing

Message Listener Configuration in Consumer-A flow

(3b) After Listening from Message Listener, we can consume as below.

Consume payload in Consumer-A flow

Consumer-B flow

(4a) Once Message is published Successfully, we can Consume the Message as below

groupId : ConsumerAB

topic: competing

Message Listener Configuration in Consumer-B flow

(4b) After Listening from Message Listener, we can consume as below.

Consume payload in Consumer-B flow

Logger to log in console

logger in Flow

Response After consumed

Tried hitting flow 3 times, as we can watch consume order is not guaranteed.

Response

Youtube Links:

Demo on Apache Kafka Pub/Sub competing consumer pattern with Mule4

Demo To create Free Kafka with Confluent Cloud

clonfluentPage for more information

For the above example please refer my git Repo

git repo: racs07/demo-kafka (github.com)

git-clone: https://github.com/racs07/demo-kafka.git