Context

Kafka is the message streaming platform of choice at Teads. We have several Kafka clusters in different regions. As of March 2021, we have 262 brokers in 22 clusters, for a total of 285 TB. Our biggest cluster has up to 32 broker nodes and contains 40 Tbyte of data.

This reduces the blast radius in case of an issue on one cluster, but increases the downside effect of the loss of a broker, compared to a big single cluster approach as done by Linkedin. Our strategy also increases the operations needed as each cluster requires dedicated time and attention.

Kafka Rebalance Nightmare

…