How can Kafka consumers parallelise beyond the number of partitions

As mentioned in my previous article, Kafka’s way of achieving parallelism is by having multiple consumers within a group. This would scale the consumers but this scaling can’t go beyond the number of partitions, as one partition can at max be assigned to one consumer in a group. One easy way of solving this problem, is by having high partitions for a topic. This might come with it’s own impacts.

We faced a similar challenge where the parallelism achieved through running multiple consumers in a group was not enough. We were looking for scaling beyond this.

Problem?

In our consumer, we were receiving the records in a stream and making an API call. In its response, we were acknowledging the record and moving to the next record.

Although the end point we were hitting in our consumer can process very high number of parallel requests, we were making only the parallel requests as many as our consumers/partitions. We were limited to making only the number of parallel requests as many as our consumers/partitions, because at any point of time we make only one request from each of our consumer.

We were not leveraging our end point which can process high number of parallel requests which is much beyond the number of parallel requests we were making.

We already had a decent number of partitions, we didn’t want to increase beyond this. When we researched, if there is any way of increasing the parallelism without increasing the number of partitions. We came across some good suggestions which actually worked for us.

Solution:

Kafka consumers parallelising beyond the number of partitions, is this even possible? Yes, we may not be able to run more number of consumers beyond the number of partitions. However, parallelism could also be achieved by parallel processing multiple records within a consumer. Apart from increasing the consumers to parallel process, we also parallel processed the records within each consumer.

There could be only one consumer thread in a consumer. However, we could spawn multiple application threads for processing those received records.

How did we implement this?

• If we have to parallel process multiple records, this means we need to receive the records in batch instead of a single record. Hence, we received the records in batch.
• Process the consumer record, in our case made the API call using application thread. We had multiple application threads to parallel process the received records.
• Make the consumer thread wait until all the API calls of the received batch are done.
• Once the entire batch is processed. Acknowledge the batch, release the consumer thread and get the next batch and process it in a similar way

One might take this approach not just to increase the parallelism beyond the number of partitions, but also to effectively utilise the consumer resources. By parallel processing within a consumer, we could achieve the same level of parallelism with lesser number of consumers.

Can we parallelise beyond the number of partitions. Not always, let’s see when:

• Before processing the first message if we can’t process the second message. With parallelism, we can’t ensure the order of message processing.
• Other system which consumer is depending on could not handle high load.

Conclusion:

We may parallelise consumers beyond the number of partitions through parallel processing of records within a consumer. However, only when the other system can also accept such high load and when there is no dependency on the order of processing.