Our contribution to the great Cruise Control Front End (CCFE) Open Source project

Context

Kafka is the message streaming platform of choice at Teads. We have several Kafka clusters in different regions. As of March 2021, we have 262 brokers in 22 clusters, for a total of 285 TB. Our biggest cluster has up to 32 broker nodes and contains 40 Tbyte of data.

This reduces the blast radius in case of an issue on one cluster, but increases the downside effect of the loss of a broker, compared to a big single cluster approach as done by Linkedin. Our strategy also increases the operations needed as each cluster requires dedicated time and attention.

Thomas Lambert

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store