Apache Kafka Guide #34 Higth Load Producer max.block.ms and buffer.memory

Apache Kafka Guide Producer max.block.ms and buffer.memory

Hi, this is Paul, and welcome to the #34 part of my Apache Kafka guide. Today we will discuss how Producer max.block.ms and buffer.memory work.

Apache Kafka At the Gates of Mastery: Kafka Core

High Load Producer

To elaborate on two key settings, namely max.block.ms and buffer.memory, let’s delve into their functionalities, especially when dealing with high throughput scenarios in Kafka producers.

If a producer is handling extremely high throughputs and encounters a scenario where the broker is unable to promptly process these requests, the data records will be temporarily stored in the producer’s memory. This buffering process relies on the buffer.memory setting, which has a default size of 32 megabytes, serving as the send buffer’s capacity.

Over time, as the broker becomes overwhelmed with requests, this buffer will gradually fill up. However, should the broker’s workload decrease, allowing it to process requests more efficiently, the buffer will then start to deplete as the data is successfully sent through, thus increasing the throughput at the broker.

The primary purpose of this buffer is to accumulate messages temporarily before they are dispatched to Kafka. This system ensures a smoother data flow, even during periods of high demand. Should the need arise due to excessive data accumulation in the buffer, there is the possibility of increasing the buffer.memory size to accommodate the surplus data.

• When the Producer produce more than the Broker can handle, the messages will be buffered in memory
• buffer.memory=33554432 (32Mb) — size of the send buffer
• The Buffer will fill up over time, and empty back down when the throughput to the Broker increases

When the buffer reaches its capacity of 32 megabytes, attempting to use the .send() method on the producer will result in a blockage. This blockage occurs at the point of executing the send command, causing the process to no longer be asynchronous. Instead of returning immediately, the operation will halt the execution of your code to prevent further filling of the buffer.

At this juncture, a new configuration is introduced. When the send method encounters a block, the `max.block.ms` setting is activated, with a value set to 60,000 milliseconds. This indicates that the system permits a blockage duration of up to 60 seconds during the send operation. If the buffer remains full beyond this period, without being cleared, an exception is thrown by the send method.

This scenario implies that the producer’s buffer is full, the broker is not accepting new data, and 60 seconds have passed. Encountering such an exception signals that the brokers might be down or excessively burdened, rendering them unable to handle any form of request. Consequently, it necessitates a thorough examination of your applications and related systems to address the issue effectively.

• When Buffer is full (32Mb) call send() method will start to block and won’t return right away
• max.block.ms=60000 — the time send() will block until throwing an Exception
• If you hit an Exception that means the Broker is down or overloaded and can’t respond to requests

Exception throwing when:

• The Producer has filled Buffer
• The Broker is not accepting new messages
• 60 sec has been elapsed

Thank you for reading until the end. Before you go:

Apache Kafka At the Gates of Mastery: Kafka Core

• Please consider clapping and following the writer! 👏
• Follow us on Twitter(X), LinkedIn