A brief description of Kafka replication mechanism

In this article, I am going to describe more about Kafka replication. It is possible whenever you see Kafka logs ask yourself what is the controller? What is the mechanism of controller and leader election? What are the things that leader and controller are responsible for? In the following, I am going to answer these types of questions. Furthermore, I will talk about the mechanism which brokers use to interact with each other in one cluster.

Review some terms in Kafka

I assume readers have some basic knowledge of Kafka like topics, partitions, producer, and consumer and I will review terms that we need them to understand the mechanism of Kafka replication:

Controller: The replica which is responsible for leader election

Leader: The replica that all requests from clients and other brokers of Kafka go to it. Each partition can only have one leader at a time.

Follower: Other replicas that are not the leader.

In-sync replica: replicas that frequently request for latest messages are considered in-sync.

Controller and its responsibilities

Kafka uses Zookeeper for replica management, when we set up a Kafka cluster, every node in the cluster requests an ephemeral node to zookeeper to be controller; the ephemeral node is a temporary node that exists until the session that created node is active. In Kafka case, all of the nodes in the cluster try to create an ephemeral node to zookeeper, since in one cluster we should have only one controller when one node created this ephemeral node, other nodes get an exception.

The controller is responsible for leader election. When we setup cluster for the first time, leaders for every partition chosen by round robin algorithm. In this way, the load is balanced over all of the brokers.

Whenever one broker leaves the cluster (in any reason), controller figures it out by watching list of nodes in zookeeper path. The controller should select another leader for that partitions that don’t have a leader. The controller simply selects the next replica in the replica list and also notifies both new leader and other brokers in the cluster.

Another responsibility that controller has is when one broker joins to cluster. In this case, the controller notifies existing brokers and also new broker to know who is the leader.

Leader and its responsibilities

Because of data consistency all requests for consuming and producing go to the leader. Followers try to be in sync with the leader by replicating messages. In-sync replicas, consistently send fetch requests to the leader; these requests are the same as consumers request that consumers send them to consume produced messages. In these requests, they send the offset that they are waiting to receive next. The producer uses a special kind of request as well. These requests are known as produce requests. Produce requests contain the message that should be sent to Kafka. Maybe you ask your self that how clients know who is the leader of the partition? Clients consistently send metadata requests to any of the brokers, in response Kafka server send all of the replicas of topics that client is interested in and their leader replicas, so clients know who is the leader of the partition, If any client sends its requests to the wrong leader it gets error response and find out should refresh its local cache by sending metadata request. This interval to send metadata request is configurable and this configuration can be set both in producer and consumer configurations.

Conclusion

In this article, I described Kafka replication briefly. Controller and leader have described and I have mentioned their main responsibilities. for example, we know that if we see these logs, what it means:

ERROR Error while creating ephemeral at /controller, node already exists and owner ‘244218314115579907’ does not match current session ‘100103126039068675’ (kafka.zk.KafkaZkClient$CheckedEphemeral)

ERROR [ReplicaFetcherThread-0–3]: Error for partition [TopicName,0] to broker 3:org. apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

The first error means that another broker tries to create an ephemeral node to be controller and second means maybe the client needs to update it’s local cache because the leader has changed.

In the next article, I will write more practical about Kafka replication.