Kafka Internal Working Mechanism

Some less explored area of Kafka

Kafka: Apache Kafka is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration and message queue system. It’s open source.

Let’s explore the unknown and in-depth facts about Kafka.

Even-if Kafka stores data on Disk how come it is fast and provides high throughput

Kafka relies heavily on the filesystem for storing and caching messages. There is a general perception that “disks are slow” which makes people skeptical that a persistent structure can offer competitive performance. In fact disks are both much slower and much faster than people expect depending on how they are used; and a properly designed disk structure can often be as fast as the network.

Kafka depends on Operating system Page Cache. Page cache is like files/data written to Disk, its index/meta cached on Ram by Operating system. Kafka takes the advantage of this. Kafka internal code gets the message from producer, then it writes to memory (page cache) and then it writes to disk. This is archived by Sequential I/O which is provided by all modern OS. You can read more about Sequential I/O on internet.

As Kafka is build on Java, you can read more about Oracle Java Documentation about Zero Copy Concept from Java side. Read about “transferTo()” method. https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html

When SSL is enabled, zero copy optimization is lost, since the Broker needs to decrypt and encrypt the data. So use SSL for Kafka wisely. But Kafka didn’t mention this straight in their documentation.

Do I need high Memory compute instance or Higher CPU cores instances for Kafka broker.

You should choose a modern processor with multiple cores. Common clusters utilize 24 core machines. In-case you are planning to use encryption in transit within your broker and application, then definitely you need to add more cores comparing to your calculated number of cores. If you need to choose between faster CPUs and More cores, choose more cores. The extra concurrency that multiple cores offers will far outweigh a slightly faster clock speed. Also avoid Network storage disk rather go with actuary attached disk like AWS EBS or something similar like attaching hard-disk to Rack where you have kept your physical server.

On consumer side, also remember if your single consumer (Pod, container, Application) is required to handle data from multiple partition, then don’t forget to give more CPUs for better read performance.

Avoid the large machines because they often lead to imbalanced resource usage. For example, all the memory is being used, but none of the CPU. Also don’t go with small machines which push you to maintain more nodes (kafka broker) in your cluster.

Why Kafka leaving Zookeeper from 4.x ?

Historically, Apache ZooKeeper was a critical part of how this distributed code worked. ZooKeeper provided the authoritative store of metadata holding the system’s most important facts: where partitions live, which replica is the leader, and so forth. The use of ZooKeeper made sense early on — it is a powerful and proven tool. For the first time, you can run Kafka without ZooKeeper starting from kafka v2.8 as preview mode. Where zookeeper will be replaced with a module called Kafka Quorum Controller based RAFT which will store all Metadata.

As part of new architecture Kafka will use new quorum controller, all metadata responsibilities previously undertaken by the Kafka controller and ZooKeeper are merged into this one new service called Quorum Controller. It is part of kafka it-self. The quorum controller can also run on dedicated VM if you have a use case that demands it.

The quorum controllers use the new KRaft protocol to ensure that metadata is accurately replicated across the quorum. This protocol is similar in many ways to ZooKeeper’s ZAB protocol. The new quorum controller is designed to handle a much larger number of partitions per cluster

Performance statistics:

Roadmap

1. Kafka can run on preview mode (experimental/non-production) without zookeeper starting version 2.8
2. From Kafka 3.x you can run kafka without zookeeper or with zookeeper (option is there)
3. From Kafka 4.x, entire zookeeper support will be deprecated ( expected as of now )

Kafka Redistribution

Even if Kafka is Opensource and Apache licensed, but there are several companies who offers Kafka on Cloud or improved/extended version of Kafka which comes with price and premium support. They maintain the original features of Kafka along with provides extended feature.

• Confluent: Confluent Cloud (SaaS, fully managed) and Confluent Platform (software download for self-managed/on-premise)
• Cloudera distribution — https://www.cloudera.com/products/open-source/apache-hadoop/apache-kafka.html
• EmblocSoft distribution and CPFA training — https://www.emblocsoft.com/pages/en/solutions/kafka/support/
• IBM Event Streams — https://www.ibm.com/cloud/event-streams — Apache Kafka on premise and the public cloud
• Strimzi — http://strimzi.io/ — Apache Kafka Operator for Kubernetes and Openshift. Downloads and Helm Chart — https://github.com/strimzi/strimzi-kafka-operator/releases/latest
• TIBCO Messaging — Apache Kafka Distribution — https://www.tibco.com/products/apache-kafka Downloads — https://www.tibco.com/products/tibco-messaging/downloads

Kafka Eco-System

Its not just installing Kafka with Zookeeper and start publishing and subscribing messages or stream data to Kafka. There is a big ecosystem which makes Kafka as on of the best message queue and streaming platform in modern Tech stack. Let’s explore the Kafka Ecosystem. Kafka provides following ecosystem.

• Kafka connect
• Kafka Stream processing
• Kafka Hadoop integration
• Kafka integration with Oracle DB
• Kafka on AWS
• Kafka on Azure
• Kafka on GCP
• Kafka logging
• GUI tools for Kafka
• Metrics for Kafka

Does Kafka repeats same message which is already delivered to same consumer ?

Well YES. was it shocking for you ?

Till kafka 2.x as zookeeper is the dependent service ( coordinator), so the Kafka works is like kafka stores metadata information in zookeeper for example each topic wise partition list and which consumer group has read up-to which offset. Also zookeeper maintainer the current leader of a partition.

But here is the catch ( Kafka does this as part of performance improvement )

Kafka does not update Zookeeper with the message offset last read after every read, instead it waits a short period of time. Due to this delay it is possible that your logic has consumed a message and that fact hasn’t been synced to zookeeper. So if your client exits/crashes you may find messages being replayed next time to start.

Also note that sometimes the loss of a Broker or other event that causes the Leader for a Partition to change can also cause duplicate messages to be replayed. So you should design your data critical application carefully.

Is there any solution to this problem ? Yes kafka do provides 2 types of consumer

1. High level consumer — This is most widely used consumer style for kafka cluster. For these style of Kafka consumers, Kafka provides techniques like Consumer group, automatic offset commit, manual offset commit, not to worry about Kafka broker (auto broker discovery), not to worry about where is the partition is (broker name), not to worry about who is the current leader of the partition. Most generic subscriber/consumer group follow this consumer style like SpringBoot, Spring, Quarkus and other programming language kafka clients.
2. Low Level consumer — This is I would say less explored and uncommon area of Kafka consumer. This don’t provides consumer group that means you as consumer in your database need to maintain last read offset, so each time your consumer is going to ask messages from kafka, consumer should provide the offset. As a consumer you decide from which partition you need to read message. As a consumer you need to findout who is the current leader of the partition (Kafka broker), if that broker goes down, you need to findout who is the new broker. So it’s kind of very heavy self managed client/consumer to deal with Kafka data subscriber part. usually extreme mission critical application need to follow this way.

Kafka Compact Topic

Can Kafka keep message forever ? The answer is Yes, Kafka can store message forever. Kafka provides topic wise configuration called “Compact topic” that means data retention of that topic will be “-1” means infinite.

As per Kafka definition: Log compaction ensures that Kafka will always retain at least the last known value for each message key within the log of data for a single topic partition. It addresses use cases and scenarios such as restoring state after application crashes or system failure, or reloading caches after application restarts during operational maintenance. For each unique message key it will retain only the last published message, other old message belongs to that key will deleted automatically.

But now the question is, evenif I am using compact type Kafka topic, lets say I want to delete a message from compact topic, is it still possible ? Answer is Yes

It’s possible to remove a message from a Kafka topic by publishing a new message with the key of the message we want to delete as null. The general tendency of kafka is that, if it finds any key with value as null, then it automatically it deletes that message. So in this way you can reduce the message size of compact topic. For this deletion kafka provides configuration as “delete.retention.ms”

Kafka Mirror Maker ( Geo replication )

Suppose you want to replicate your Kafka cluster data to another Kafka cluster then you need something called MirrorMaker. Usually for Disaster recovery this tool has been used.

Kafka’s mirroring feature makes it possible to maintain a replica of an existing Kafka cluster. MirrorMaker tool to mirror a source Kafka cluster into a target (mirror) Kafka cluster. The tool uses a Kafka consumer to consume messages from the source cluster, and re-publishes those messages to the local (target) cluster using an embedded Kafka producer.

Kafka administrators can define data flows that cross the boundaries of individual Kafka clusters, data centers, or geo-regions. With MirrorMaker you can monitor the status of Sync based on single topic or all topic just to monitor how replication is working.

When you should starting thinking about MirrorMake. Following are the high level use cases and need.

• Geo-replication
• Disaster recovery
• Feeding edge clusters into a central, aggregate cluster
• Physical isolation of clusters (such as production vs. testing)
• Cloud migration or hybrid cloud deployments

MirrorMaker has way more configuration than you expect ( if you are new) . Let’s consider A being Kafka cluster-1 and B the 2nd Kafka cluster (same for C, D and K cluster). Following sync configuration is possible between multiple Kafka clusters.

• 2 way sync between cluster ( Active-Active setup) : A->B, B->A
• 1 Way Sync ( Active-Passive setup) : A->B
• Aggregation (from many clusters to one): A->K, B->K, C->K
• Fan-out (from one to many clusters): K->A, K->B, K->C
• Chaining or Forwarding: A->B, B->C, C->D

MirrorMaker shell script “connect-mirror-maker.sh”, you can find inside “bin” folder of latest Kafka distribution.

Thanks for reading. Don’t forget to like this and share.