Things nobody will tell you setting up a Kafka cluster

Setting up a Kafka cluster for a production environment is no trivial task. There are so many configurations, settings, and decisions to be taken that it’s quite difficult to understand all of them up front. Even though after going through entire confluent documentation and reading everything available about the best practices, you are bound to make mistakes and spend days, if not weeks pondering over what is going wrong and why is the cluster so unstable.

Documentation provided by confluent is indeed very good and contains everything you’ll ever need to set up a cluster. But sometimes you do miss a few things and end up learning them the hard way.

In this article, I am going to discuss some of the key points to keep in mind when setting up a Kafka cluster. Few things which we missed and had a hard time fixing up and some other things that I feel anybody can miss when setting up a cluster for the first time.

I am assuming that you already know the basics of Kafka and generally understand how to set up a cluster with Zookeeper and all.

Points :

  • Set vm.swappiness to the minimum possible value for your server type for both Kafka brokers as well as the zookeeper servers.
    Swappiness is basically Linux kernel's tendency to copy RAM contents to swap. For latency critical application such as Kafka, it's very important that unnecessary swapping of memory contents doesn’t affect its performance. You can go here to know how to do it in Ubuntu.

We just walked through a very small set of key points that one must keep in mind when setting up a Kafka cluster. I am pretty sure that you’ll face many other weird issues of your own which will make you scratch your head just like we did. Hopefully, this article will help you avoid some of those.




Be careful what you wish for

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store