Understanding Kafka Topic Partitions
Everything in Kafka is modeled around partitions. They rule Kafka’s storage, scalability, replication, and message movement.
--
P.S I did some edits to reflect the feedback I received from the audience. Thanks for your valuable contribution. I expect more! :)
Everything in Kafka revolves around partitions. They play a crucial role in structuring Kafka’s storage and the production and consumption of messages.
Understanding partitions helps you learn Kafka faster. This article walks through the concepts, structure, and behavior of Kafka’s partitions.
Events, Streams, and Kafka Topics
Before diving into partitions, we need to set the stage here. So let’s look at some high-level concepts and how they relate to partitions.
Events
An event represents a fact that happened in the past. Events are immutable and never stay in one place. They always travel from one system to another system, carrying the state changes that happened.
Streams
An event stream represents related events in motion.
Topics
When an event stream enters Kafka, it is persisted as a topic. In Kafka’s universe, a topic is a materialized event stream. In other words, a topic is a stream at rest.
Topic groups related events together and durably stores them. The closest analogy for a Kafka topic is a table in a database or folder in a file system.
Topics are the central concept in Kafka that decouples producers and consumers. A consumer pulls messages off of a Kafka topic while producers push messages into a Kafka topic. A topic can have many producers and many consumers.
Partitions
Kafka’s topics are divided into several partitions. While the topic is a logical concept in Kafka, a partition is the smallest storage unit that holds a subset of records owned by a topic. Each partition is a single log file where records are written to it in an append-only fashion.
When talking about the content inside a partition, I will use the terms record and message interchangeably.
Offsets and the ordering of messages
The records in the partitions are each assigned a sequential identifier called the offset, which is unique for each record within the partition.
The offset is an incremental and immutable number, maintained by Kafka. When a record is written to a partition, it is appended to the end of the log, assigning the next sequential offset. Offsets are particularly useful for consumers when reading records from a partition. We’ll come to that at a later point.
The figure below shows a topic with three partitions. Records are being appended to the end of each one.
Although the messages within a partition are ordered, messages across a topic are not guaranteed to be ordered.
Partitions are the way that Kafka provides scalability
A Kafka cluster is made of one or more servers. In the Kafka universe, they are called Brokers. Each broker holds a subset of records that belongs to the entire cluster.
Kafka distributes the partitions of a particular topic across multiple brokers. By doing so, we’ll get the following benefits.
- If we are to put all partitions of a topic in a single broker, the scalability of that topic will be constrained by the broker’s IO throughput. A topic will never get bigger than the biggest machine in the cluster. By spreading partitions across multiple brokers, a single topic can be scaled horizontally to provide performance far beyond a single broker’s ability.
- A single topic can be consumed by multiple consumers in parallel. Serving all partitions from a single broker limits the number of consumers it can support. Partitions on multiple brokers enable more consumers.
- Multiple instances of the same consumer can connect to partitions on different brokers, allowing very high message processing throughput. Each consumer instance will be served by one partition, ensuring that each record has a clear processing owner.
Partitions are the way that Kafka provides redundancy.
Kafka keeps more than one copy of the same partition across multiple brokers. This redundant copy is called a replica. If a broker fails, Kafka can still serve consumers with the replicas of partitions that failed broker owned.
Partition replication is complex, and it deserves its own post. Next time maybe?
Writing records to partitions.
How does a producer decide to which partition a record should go? There are three ways a producer can rule on that.
Using a partition key to specify the partition
A producer can use a partition key to direct messages to a specific partition. A partition key can be any value that can be derived from the application context. A unique device ID or a user ID will make a good partition key.
By default, the partition key is passed through a hashing function, which creates the partition assignment. That assures that all records produced with the same key will arrive at the same partition. Specifying a partition key enables keeping related events together in the same partition and in the exact order in which they were sent.
Key based partition assignment can lead to broker skew if keys aren’t well distributed.
For example, when customer ID is used as the partition key, and one customer generates 90% of traffic, then one partition will be getting 90% of the traffic most of the time. On small topics, this is negligible, on larger ones, it can sometime take a broker down.
When choosing a partition key, ensure that they are well distributed.
Allowing Kafka to decide the partition
If a producer doesn’t specify a partition key when producing a record, Kafka will use a round-robin partition assignment. Those records will be written evenly across all partitions of a particular topic.
However, if no partition key is used, the ordering of records can not be guaranteed within a given partition.
The key takeaway is to use a partition key to put related events together in the same partition in the exact order in which they were sent.
Writing a custom partitioner
In some situations, a producer can use its own partitioner implementation that uses other business rules to do the partition assignment.
Reading records from partitions.
Unlike the other pub/sub implementations, Kafka doesn’t push messages to consumers. Instead, consumers have to pull messages off Kafka topic partitions. A consumer connects to a partition in a broker, reads the messages in the order in which they were written.
The offset of a message works as a consumer side cursor at this point. The consumer keeps track of which messages it has already consumed by keeping track of the offset of messages. After reading a message, the consumer advances its cursor to the next offset in the partition and continues. Advancing and remembering the last read offset within a partition is the responsibility of the consumer. Kafka has nothing to do with it.
By remembering the offset of the last consumed message for each partition, a consumer can join a partition at the point in time they choose and resume from there. That is particularly useful for a consumer to resume reading after recovering from a crash.
A partition can be consumed by one or more consumers, each reading at different offsets.
Kafka has the concept of consumer groups where several consumers are grouped to consume a given topic. Consumers in the same consumer group are assigned the same group-id value.
The consumer group concept ensures that a message is only ever read by a single consumer in the group.
When a consumer group consumes the partitions of a topic, Kafka makes sure that each partition is consumed by exactly one consumer in the group.
The following figure shows the above relationship.
Consumer groups enable consumers to parallelize and process messages at very high throughputs. However, the maximum parallelism of a group will be equal to the number of partitions of that topic.
For example, if you have N + 1 consumers for a topic with N partitions, then the first N consumers will be assigned a partition, and the remaining consumer will be idle, unless one of the N consumers fails, then the waiting consumer will be assigned its partition. This is a good strategy to implement a hot failover.
The figure below illustrates this.
The key takeaway is that number of consumers don’t govern the degree of parallelism of a topic. It’s the number of partitions.
