An investigation into Kafka Log Compaction

Luke Stephenson
Apr 8, 2020 · 10 min read
https://kafka.apache.org/

Background

At Zendesk, we have been using Kafka in production for several years. Primarily we have been using it to distribute events between services. For example, “a ticket has been changed, and this is what changed”. We are now considering using Kafka to share entities between systems. For example, each time a ticket is updated we will publish the full latest version of that…