Why should anyone use Apache Kafka?

Apache Kafka, hereafter addressed as just Kafka for convenience(not to be confused with the famous author who actually inspired the name for the technology), originated as a distributed streaming platform within LinkedIn and later open sourced to the world, is now the backbone of every enterprise that makes sense of the huge volumes of data they both acquire from users and generate as part of doing normal business in the connected world.

Why send messages across systems?

As systems grow and evolve, they break out into larger number of disparate components, both for improving resilience of the entire system(so error in one component doesn’t bring the entire system down) as well as to handle the increased load within the performance expectations of the system(a fixed set of machines can only perform a certain number of operations in a given time window, so they eventually scale out to meet the increasing load). The original system is thus decomposed into smaller sub-systems(some take it to the extreme with microservices and nanoservices) and in other cases these were disparate systems to begin with(operational and analytical systems). However, it is not enough to break down the processing logic into separate components and deploy them independently. Each of these subsystems also needs to have access to the data to do the intended…