A high throughput multi-threaded ordered Kafka consumer
When events are being consumed from a Kafka topic which has multiple partitions and high volume of generated events, the slow speed of consumption could cause high consumer lag . A typical strategy to increase the throughput of consumption is to scale out and have multiple consumers reading concurrently through different partitions.
This approach however has limitation, for a topic which has high number of partitions scaling the no of consumer so that each partition is running on a separate consumer/node is not a good strategy. Even in scenarios where the number of partitions are limited the processing latency on the Kafka consumer thread can cause the consumer lag. This can typically happen in a micro services environment where Kafka consumer logic is making call to the downstream service to do some action and if the downstream service call is not asynchronous the slow downstream can cause the event processing to get stalled.
A common solution to the Kafka consumer throughput problem is to have a single threaded consumer which delegates the processing of the event post consumption to a thread pool which asynchronously handles the event processing. The main…