Kafka Unclean Leader Election

The unclean leader election configuration is used to determine whether a replica that is not in-sync with the lead replica can itself become leader in a failure scenario. However if this were to happen, any messages that the unclean leader did not have would be lost.

It is therefore important to understand the consequences for the system based on how this parameter is configured, and the associated trade-offs that must be evaluated.

Availability vs Durability

Consider the following Kafka Broker state, with three Broker nodes. The diagram shows that node 1 is the lead replica for a particular topic partition, and the three messages written to the partition, namely ‘foo’, ‘bar’ and ‘xyz’, are being replicated to nodes 2 and 3.

When unclean leader election is enabled the following scenario results in message loss:

Allowing the unclean broker to become leader as a last resort increases availability at the expense of durability, as new reads and…