KAFKA : producer & consumer C#(part 1)
Most of my stories oriented for practical exercise (you can see my previous story), but before start to working with Kafka Consumer & Producer C#, it’s better understand what is exactly Kafka, broker, topic, partition, producer, message, consumer, and etc.
What is Kafka?
Apache Kafka is a publish | subscribe messaging system designed to solve many issues. In many articles you can see described info as a distributed commit log or distributing streaming platform, which scalable, asynchronous and a fault tolerant.
Kafka Broker | Kafka Server | Kafka Node
Broker is single Kafka service. Main idea it’s receive messages from producers, assigned offsets to them, commits the messages to storage on disk. Also broker takes care about services consumers, responding to fetch requests for partitions and responding with the messages that have been committed to disk.
Topics and Partitions
Messages in Kafka are categorized into topics. The simple analog for a topic it’s a table in database or a folder in a filesystem. Topics are additionally broken down into a number of partitions. Going back to the “commit log” description, a partition is a single log. Messages are written to it in an append-only fashion, and are read in order from beginning to end.
Attention : Topic which has multiple partitions, doesn’t guarantee of message time-ordering across the entire topic, just within a single partition.
Producer is client application that publish | write events to Kafka.
Consumer is application that subscribe to read and process the events. In our modern world with many dev languages, consumers and producers can be written in different languages.
Kafka Message | Record | Event (key — value pair)
The unit of data within Kafka is called a message. if you have experience with database background, it’s possible to draw parallel with a row or a record. A message is simply an array of bytes as far as Kafka is concerned, so the data contained within it does not have a specific format or meaning to Kafka. A message can have an optional bit of metadata, which is referred to as a key. The key is also a byte array and, as with the message, has no specific meaning to Kafka. Keys are used when messages are to be written to partitions in a more controlled manner. The simplest such scheme is to generate a consistent hash of the key, and then select the partition number for that message by taking the result of the hash modulo, the total number of partitions in the topic. This assures that messages with the same key are always written to the same partition (will comeback in demo part). Also the message include fields : key, value, timestamp, headers.
Consumer Group
Each Kafka Consumer is part of a consumer group. Same consumer group allow multiple consumers to read from the same topic, splitting the data between them. When multiple consumers are subscribed to a topic and belong to the same consumer group, each consumer in the group will receive messages from a different subset of the partitions in the topic.
In case when one consumer group has more consumers then partitions in topic, some of the consumers will be idle and get no messages at all.
Order messages in Kafka topic
In Kafka topic with one partition order can be guaranteed. On side of producer all messages what was send with one order, it will be written them to partition. On consumer side, it will be read with the same order.
It’s easy. Let’s see visual example.
But situation can be changed for Topic with multiple partitions.
There are several methods :
Round Robin (default) method — when messages will be send in different partition, ensuring that overloading isn’t happen on one of partition.
Hashing Key Partition — during producing message in record specify same key for group of messages. it will ensure that all of them will go in the same partition.
Custom Partitioner — providing own mechanism to send data in specified partition (will not see in this article).
Demo time
We need the following tools installed on our computer:
If you are ready, let’s get started.
docker-compose file was reused from previous article KAFKA in docker container and command line. Create kafka docker container and execute command
# go inside container
docker container exec -it kafka /bin/bash
# create kafka topic with two partitions.
kafka-topics --bootstrap-server localhost:9092 --topic users-in --create --partitions 2 For demo was created simple Web API project with endpoints.
Add a JsonKafkaProducer, KafkaBackgroundService (as Consumer) and Model
Now, we will create Kafka Producer and Consumer also our data model classes.
In Solution Explorer, right-click the project. Select Add -> New Folder and name the folder Services.
Then right-click the Services folder and select Add->Class. Name the class JsonKafkaProducer.cs and click Add.
Also create interface IKafkaProducer.cs
In same Service folder create KafkaBackgroundService.cs
FYI: For Demo it was decided to implement KafkaConsumer and JsonKafkaProducer in one API. But in modern microservice architecture it can be in provided different microservices. Code Examples can be adapted and for specific requirements of business.
then In Solution Explorer -> Add -> New Folder and name the folder Entities.
Then right-click the Entities folder and Add->Class. Name the class UserEntityRequest.cs and click Add.
Next, add the following properties to the class:
In the Program.cs class of our project register JsonKafkaProducer, KafkaBackgroundService and etc:
in Controller folder create KafkaController.cs
At this point we have a starter project as follows:
Now, let’s run (Ctlr+F5) the project to see the default output. When the browser opens and the Swagger UI is shown
select the GET method in the KafkaController part and then select Try It Out and Execute:
[{
"username": "Bret",
"email": "Sincere@april.biz"
},
{
"username": "Antonette",
"email": "Shanna@melissa.tv"
}]also try to Execute : with one object
[{
"username": "Bret",
"email": "Sincere@april.biz"
}]FYI : on https://json-generator.com/ site you can generate JSON. Just Copy&Paste template bellow.
[
'{{repeat(5, 7)}}',
{
username: '{{firstName()}} {{surname()}}',
email: '{{email()}}'
}
]Open in browser new tab and type localhost:3040 -> click enter Topics -> choose users-in topic.
During producing message in kafka topic with one partition it’s possible to guarantee ordering of messages. But what to do if you have more than 1 partitions in topic. There is solution you can guarantee ordering messages in one partition.
Specify same key for group of messages, so that all messages on the same key go to the same partition is very important for proper ordering of message processing if you will have multiple consumers in a consumer group on a topic.
This is the end for current post. You can find the full project in GitHub.
Hope you found this post helpful and easy to follow. Please let me know if you have any corrections and/or questions in the comments below.
And if you liked this post, please 👏👏👏 your hands or buy me coffee.
To be Continue
====> KAFKA: PostgreSql to Kafka with using KafkaConnect (part 2)
References
