What’s new in Kafka 3.8
At the end of July 2024, according to the current project development plan, a universal and powerful tool for building real-time data pipelines and event-driven applications Apache Kafka 3.8 was released with JBOD support in multi-tier storage, detailed compression level settings, security improvements and ability to remove ambiguous configurations. The previous version of Apache Kafka 3.7 was released in February 2024. The next release of Apache Kafka 3.9 is planned for early autumn.
The project developers have updated the documentation section to upgrade to 3.8.0 from any version 0.8.x to 3.7.x with detailed steps for performing the actions correctly.
Major changes and new features in Apache Kafka 3.8:
- JBOD (Just a Bunch Of Disks) support in KRaft-based clusters has been moved to production status. With full JBOD support, you can maintain multiple log directories per broker to run large deployments with various storage devices on different nodes of a Kafka cluster with the Kafka protocol;
- Producer, broker, and topic configuration mechanism for specifying the compression level has been enabled, and three new integer parameters have been added for the gzip, lz4, and zstd codecs: compression.gzip.level, compression.lz4.level, and compression.zstd.level. The snappy codec has been deprecated as it does not support any compression level;
- Added the ability to repeat the download process when updating metadata if no known brokers are available;
- Added the ability to limit files accessible to file and directory configuration providers to avoid unrestricted access to improve security. Also added the remote.fetch.max.wait.ms configuration — maximum remote fetch timeout for DelayedRemoteFetch requests when reading from remote storage. This allows you to customize the timeout based on your workload;
- Kafka Streams now features state store sharing, which allows data in a state store to be used across multiple applications without topic-level duplication, and customizable task assignment for streams, which replaces the existing internal assignor task configuration. Several new metrics have also been added to help find memory leaks and performance issues in Kafka Streams applications;
- Kafka Connect has changed the behavior of the tasks.max.enforce property, which when set to false allowed the connector to generate more than the maximum number of tasks. However, connectors that generate too many tasks fail, as do existing task sets that exceed the tasks.max property. To prevent the risk of this error, the tasks.max.enforce property is deprecated in Kafka 3.8 and will be removed in the next major release.
You can say Hi to us at Pharos Production — a software development company
Follow our product Ludo — the reputational system of the Web3 world
