Kafka 3.6.0 릴리즈

Kafka Release 3.6.0

Kafka Version 3.6.0 Release

Apache Kafka 3.6.0 버전이 릴리즈되었습니다.

어떠한 부분이 개선되고 변경되었는지 간단하게 안내해 드릴게요.

새로운 기능

• [KAFKA-7739] — Kafka Tiered Storage
• [KAFKA-14305] — KRaft Metadata Transactions
• [KAFKA-14627] — Modernize Connect plugin discovery
• [KAFKA-15030] — Add connect-plugin-path command line tool
• [KAFKA-15031] — Add plugin.discovery worker configuration
• [KAFKA-15228] — Add sync-manifests subcommand to connect-plugin-path tool

이번 업데이트에서는 Tiered Storage의 Early access 기능이 추가되었어요.

Kafka의 사용량이 많아질수록 스토리지의 중요성도 증가해요. Kafka를 통해 처리되는 데이터가 많아질수록 스토리지의 사용량도 늘어나고 이에 따른 높은 성능과 안정성도 동시에 충족되어야 하기 때문이에요.

이번 Tiered Storage 기능에서 제공하는 계층형 스토리지 접근 방식을 통해 메세지의 보존 기간을 로컬 스토리지 계층과 원격 스토리지 계층에서 달리 설정하여 카프카의 효율성을 높이고 안정적으로 활용할 수 있어요.

Tiered Storage에 대한 더 상세한 정보는 아래 링크를 통해 확인할 수 있어요.

스피타 기술블로그 Tiered Storage 개요

Kafka tiered storage early access release notes

이번 Tiered Storage(KIP-405) 기능은 Early access 버전이기 때문에 중요성이 높은 운영(Production)환경에서 이 기능을 활용하시는 것은 아직 권장드리지 않습니다.

개선 사항

• [KAFKA-4107] — Support offset reset capability in Kafka Connect
• [KAFKA-8982] — Admin.deleteRecords should retry when failing to fetch metadata
• [KAFKA-12261] — Splitting partition causes message loss for consumers with auto.offset.reset=latest
• [KAFKA-13299] — Accept listeners that have the same port but use IPv4 vs IPv6
• [KAFKA-13431] — Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)
• [KAFKA-13504] — Retry connect internal topics’ creation in case of InvalidReplicationFactorException
• [KAFKA-13875] — update docs to include topoicId for kafka-topics.sh — describe output
• [KAFKA-14038] — Optimize calculation of size for log in remote tier
• [KAFKA-14539] — Simplify StreamsMetadataState by replacing the Cluster metadata with partition info map
• [KAFKA-14661] — Upgrade Zookeeper to 3.8.2
• [KAFKA-14669] — Include MirrorMaker connector configurations in docs
• [KAFKA-14709] — Move content in connect/mirror/README.md to the docs
• [KAFKA-14735] — Improve KRaft metadata image change performance at high topic counts
• [KAFKA-14752] — improve kafka examples under examples package
• [KAFKA-14766] — Improve performance of VarInt encoding/decoding
• [KAFKA-14791] — Create a builder class for PartitionRegistration
• [KAFKA-14828] — Remove R/W lock from StandardAuthorizer
• [KAFKA-14866] — When broker shutdown, the controller module needs to remove its metrics
• [KAFKA-14868] — Remove some forgotten metrics when the replicaManager is closed
• [KAFKA-14926] — Remove metrics on Log Cleaner shutdown
• [KAFKA-14936] — Add Grace Period To Stream Table Join
• [KAFKA-14937] — Refactoring for client code to reduce boilerplate
• [KAFKA-14944] — Reduce CompletedFetch#parseRecord() memory copy
• [KAFKA-14982] — Improve the kafka-metadata-quorum output
• [KAFKA-14988] — Upgrade scalaCollectionCompact to v2.9 for CVE-2022–36944
• [KAFKA-14991] — Improving Producer’s record timestamp validation
• [KAFKA-14993] — Improve TransactionIndex instance handling while copying to and fetching from RSM.
• [KAFKA-15034] — Improvement of ReplaceField performance for long list
• [KAFKA-15036] — Kraft leader change fails when invoking getFinalizedFeatures
• [KAFKA-15039] — Reduce logging level to trace in PartitionChangeBuilder.tryElection()
• [KAFKA-15076] — KRaft should prefer snapshots when listeners are at the start of the log
• [KAFKA-15078] — When fetching offset 0 the KRaft leader should response with SnapshotId
• [KAFKA-15085] — Make Timer.java implement AutoCloseable
• [KAFKA-15107] — Additional custom metadata for remote log segment
• [KAFKA-15121] — FileStreamSourceConnector and FileStreamSinkConnector should implement KIP-875 APIs (alterOffsets)
• [KAFKA-15123] — Add tests for ChunkedBytesStream
• [KAFKA-15126] — Change range queries to accept null lower and upper bounds
• [KAFKA-15130] — Delete remote segments when delete a topic
• [KAFKA-15131] — Improve RemoteStorageManager exception handling documentation
• [KAFKA-15139] — Optimize the performance of `Set.removeAll(List)` in `MirrorCheckpointConnector`
• [KAFKA-15141] — High CPU usage with log4j2
• [KAFKA-15153] — Use Python `is` instead of `==` to compare for None
• [KAFKA-15155] — Follow PEP 8 best practice in Python to check if a container is empty
• [KAFKA-15159] — Update minor dependencies in preparation for 3.5.1
• [KAFKA-15177] — MirrorMaker 2 should implement the alterOffsets KIP-875 API
• [KAFKA-15182] — Normalize offsets before invoking SourceConnector::alterOffsets
• [KAFKA-15183] — Add more controller, loader, snapshot emitter metrics
• [KAFKA-15213] — Provide the exact offset to QuorumController.replay
• [KAFKA-15219] — Support delegation tokens in KRaft
• [KAFKA-15222] — Upgrade zinc scala incremental compiler plugin version to a latests stable fit version (1.9.2)
• [KAFKA-15245] — Improve Tiered Storage Metrics
• [KAFKA-15291] — Implement Versioned interfaces in common Connect plugins
• [KAFKA-15336] — Connect plugin Javadocs should mention serviceloader manifests

Bug Fixes

• [KAFKA-8690] — Flakey test ConnectWorkerIntegrationTest#testAddAndRemoveWorke
• [KAFKA-9926] — Flaky test PlaintextAdminIntegrationTest.testCreatePartitions
• [KAFKA-10337] — Wait for pending async commits in commitSync() even if no offsets are specified
• [KAFKA-10579] — Flaky test connect.integration.InternalTopicsIntegrationTest.testStartWhenInternalTopicsCreatedManuallyWithCompactForBrokersDefaultCleanupPolicy
• [KAFKA-12525] — Inaccurate task status due to status record interleaving in fast rebalances in Connect
• [KAFKA-12842] — Failing test: org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest.testSourceTaskNotBlockedOnShutdownWithNonExistentTopic
• [KAFKA-13197] — KStream-GlobalKTable join semantics don’t match documentation
• [KAFKA-13337] — Scanning for Connect plugins can fail with AccessDeniedException
• [KAFKA-13668] — Failed cluster authorization should not be fatal for producer
• [KAFKA-14273] — Kafka doesn’t start with KRaft on Windows
• [KAFKA-14654] — Connectors have incorrect Thread Context Classloader during initialization
• [KAFKA-14662] — ACL listings in documentation are out of date
• [KAFKA-14694] — RPCProducerIdManager should not wait for a new block
• [KAFKA-14712] — Confusing error when writing downgraded FeatureImage
• [KAFKA-14831] — Illegal state errors should be fatal in transactional producer
• [KAFKA-14863] — Plugins which do not have a valid no-args constructor are visible in the REST API
• [KAFKA-14938] — Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary
• [KAFKA-14962] — Whitespace in ACL configuration causes Kafka startup to fail
• [KAFKA-14967] — MockAdminClient throws NullPointerException in CreateTopicsResult
• [KAFKA-14978] — ExactlyOnceWorkerSourceTask does not remove parent metrics
• [KAFKA-14997] — JmxToolTest failing with initializationError
• [KAFKA-15012] — JsonConverter fails when there are leading Zeros in a field
• [KAFKA-15016] — LICENSE-binary file contains dependencies not included anymore
• [KAFKA-15021] — KRaft controller increases leader epoch when shrinking ISR
• [KAFKA-15053] — Regression for security.protocol validation starting from 3.3.0
• [KAFKA-15059] — Exactly-once source tasks fail to start during pending rebalances
• [KAFKA-15077] — FileTokenRetriever doesn’t trim the token before returning it.
• [KAFKA-15080] — Fetcher’s lag never set when partition is idle
• [KAFKA-15091] — Javadocs for SourceTask::commit are incorrect
• [KAFKA-15096] — CVE 2023–34455 — Vulnerability identified with Apache kafka
• [KAFKA-15098] — KRaft migration does not proceed and broker dies if authorizer.class.name is set
• [KAFKA-15100] — Unsafe to call tryCompleteFetchResponse on request timeout
• [KAFKA-15102] — Mirror Maker 2 — KIP690 backward compatibility
• [KAFKA-15106] — AbstractStickyAssignor may stuck in 3.5
• [KAFKA-15109] — ISR shrink/expand issues on ZK brokers during migration
• [KAFKA-15114] — StorageTool help specifies user as parameter not name
• [KAFKA-15135] — RLM listener configurations passed but ignored by RLMM
• [KAFKA-15137] — Don’t log the entire request in KRaftControllerChannelManager
• [KAFKA-15145] — AbstractWorkerSourceTask re-processes records filtered out by SMTs on retriable exceptions
• [KAFKA-15162] — Reflective plugin scanning misses plugins which are in parent classloaders but not classpath
• [KAFKA-15189] — Do not initialize RemoteStorage related metrics when disabled at cluster
• [KAFKA-15212] — Remove unneeded classgraph license file
• [KAFKA-15216] — InternalSinkRecord::newRecord method ignores the headers argument
• [KAFKA-15218] — NPE will be thrown while deleting topic and fetch from follower concurrently
• [KAFKA-15220] — KRaftMetadataCache returns fenced brokers from getAliveBrokerNode
• [KAFKA-15235] — No test coverage reports for Java due to settings for Jacoco being incompatible with Gradle 8.x
• [KAFKA-15238] — Connect workers can be disabled by DLQ-related blocking admin client calls
• [KAFKA-15243] — User creation mismatch
• [KAFKA-15244] — Connect PluginType.from(Class) result is incorrect when subclassing multiple plugin interfaces
• [KAFKA-15263] — KRaftMigrationDriver can run the migration twice
• [KAFKA-15312] — FileRawSnapshotWriter must flush before atomic move
• [KAFKA-15319] — Upgrade rocksdb to fix CVE-2022–37434
• [KAFKA-15338] — The metric group documentation for metrics added in KAFKA-13945 is incorrect
• [KAFKA-15345] — KRaft leader should notify the listener only when it has read up to the leader’s epoch
• [KAFKA-15353] — Empty ISR returned from controller after AlterPartition request
• [KAFKA-15374] — ZK migration fails on configs for default broker resource
• [KAFKA-15375] — When running in KRaft mode, LogManager may creates CleanShutdown file by mistake
• [KAFKA-15377] — GET /connectors/{connector}/tasks-config endpoint exposes externalized secret values
• [KAFKA-15389] — MetadataLoader may publish an empty image on first start
• [KAFKA-15391] — Delete topic may lead to directory offline
• [KAFKA-15404] — Failing Test DynamicBrokerReconfigurationTest#testThreadPoolResize
• [KAFKA-15414] — remote logs get deleted after partition reassignment
• [KAFKA-15429] — Kafka Streams attempts to commit on a closed producer when shutting down after an exception when running with EOS
• [KAFKA-15435] — KRaft migration record counts in log message are incorrect
• [KAFKA-15441] — Broker sessions can time out during ZK migration
• [KAFKA-15450] — Disable ZK migration when JBOD configured
• [KAFKA-15473] — Connect connector-plugins endpoint shows duplicate plugins
• [KAFKA-15487] — CVE-2023–40167, CVE-2023–36479 — Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1
• [KAFKA-15498] — Upgrade Snappy-Java to 1.1.10.4
• [KAFKA-15503] — CVE-2023–40167, CVE-2023–36479 — Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1

더 자세한 정보는 아래 Release Notes를 참고하세요

downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html