Not so great kafka configuration and how I fixed it
Kafka is really great, it’s probably one of the most exciting open source projects in the whole IT industry. But it also has its own quirks…
If you are working with kafka a lot, you must be dealing with consumers, producers or admin clients. For example, in order to create new kafka consumer, you need to provide some configuration to it. KafkaConsumer class constructor accepts only java.util.Properties or Map<String, Object> as it’s configuration. So how you should know what values you are able to pass?
Just look at ConsumerConfig class you might say! Strangely, instead of being a real config class, with exposed fields or properties, it is basically just a placeholder for all consumer config constants which you could use:
You can also notice quite strange way to add documentation to configuration keys by having private constants and naming them using DOC suffix.
You might think that this is fine, until you try to add some configuration:
And then running the test…
org.apache.kafka.common.config.ConfigException: Invalid value 60000 for configuration request.timeout.ms: Expected value to be a 32-bit integer, but it was a java.lang.Long
Duration.toMillis returns long and request.timeout.ms expects to get int. But how should I know this? Of course you could read all the source code from kafka-clients library and then probably you’ll eventually find out, that for some configuration values where milliseconds should be provided, you need to pass ints and for other ones — longs (e.g. retry.backoff.ms expects long as a value). I would really like understand logical reason for all of this stuff but I guess I’ll never find out :)
So maybe we could do a bit better?
I’ve just released my new kafka-config library on github which should help you to avoid these issues. It basically provides type-safe builders for constructing configuration values for kafka consumers, producers and admin clients. So instead of putting some random keys and values, you can now build your configuration fluently:
Actually, all configuration values for consumers, producers, etc. are available by using the idiomatic data types in Java and you get javadoc for free:
It will be much harder to mess up your kafka configuration now :)
