Apache Kafka Guide #24 Large Messages

Hi, this is Paul, and welcome to the #24 part of my Apache Kafka guide. Today we will discuss how to exchange large messages using Apache Kafka.

Apache Kafka At the Gates of Mastery: Kafka Core

This topic will examine how to transmit sizable messages using Apache Kafka. Kafka typically allows messages up to one megabyte in topics, which is a sensible default. Messages larger than this are often seen as inefficient and not recommended. However, two methods are available if you must send bigger messages in Apache Kafka.

External Storage

Firstly, you can utilize an external storage solution. This involves placing your message in HDFS, Amazon S3, Google Cloud Storage, or any FTP of your choice, and then sending a link to this message to Apache Kafka.

Increase the Kafka message size

The second option involves enhancing Kafka’s capacity to accommodate larger messages. Specifically, we’re looking at sending messages up to 10Mb directly from the producer through the topic to the consumer. On the Kafka side, for the topic, you need to adjust a broker setting named message.max.bytes. To modify the topic side, use max.message.bytes. This naming convention may seem confusing as the terms are inverted, but it’s intentional and not a mistake. The names and effects are similar but distinct.

Additionally, on the broker side, you can set the maximum replication fetch size to 10Mb. This setting is known as replica.fetch.max.bytes and is found in the server.properties file, which you need to edit. On the consumer side, it’s crucial to increase the fetch size to prevent crashes. This is done by setting max.partition.fetch.bytes it to 10Mb. Likewise, on the producer side, adjust the max.request.size to 10Mb. Implementing all these settings allows the transmission of larger messages in Apache Kafka.

Topic-wise, Kafka-side, change message size up to 10Mb:

• Broker side: set message.max.bytes
• Topic side: set max.message.bytes

Broker-wise, change max replication fetch size up to 10Mb:

• replica.fetch.max.bytes=10485880 (server.properties file)

Consumer-side, change fetch size:

• max.partition.fetch.bytes=10485880

Producer-side, change max request size:

• max.request.size=10485880

Thank you for reading until the end. Before you go:

Apache Kafka At the Gates of Mastery: Kafka Core

• Please consider clapping and following the writer! 👏
• Follow us on Twitter(X), LinkedIn