Kafka 3.7.0 릴리즈

Kafka Release 3.7.0

Kafka Version 3.7.0 Release

Apache Kafka 3.7.0 버전이 릴리즈되었습니다.

어떠한 부분이 개선되고 변경되었는지 간단하게 안내해 드릴게요.

New Feature

• [KAFKA-9800] — [KIP-580] Client Exponential Backoff Implementation
• [KAFKA-15215] — The default.dsl.store config is not compatible with custom state stores
• [KAFKA-15428] — Cluster-wide dynamic log adjustments for Connect
• [KAFKA-15445] — KIP-975: Docker Image for Apache Kafka
• [KAFKA-15448] — Streams StandbyTaskUpdateListener
• [KAFKA-15470] — Allow creating connectors in a stopped state

이번 업데이트부터는 KIP-975에서 다뤄졌던 Apache Kafka의 Docker 이미지 지원이 공식화되었습니다.

Apache Kafka는 그간 Docker 이미지가 지원되지 않았었습니다. 따라서 Kubernetes 환경에서 Kafka를 사용하고자 하는 사용자들은 스스로 컨테이너 이미지를 만들거나, bitnami와 같이 별도로 배포되는 이미지를 활용해야 했어요. 하지만 이제부터는 매 Kafka 새로운 버전이 릴리즈될 때마다 Docker 이미지(Scala 2.13만 지원)도 함께 배포될 예정이에요. Proposed Changes는 아래와 같아요

• 아파치 카프카의 각 릴리스마다 도커 이미지가 추가 아티팩트로 제공됩니다.
• 이 도커 이미지는 JVM 기반 카프카로, 리눅스 기반 AMD 및 ARM 아키텍처를 지원할 것입니다.
• 각 태그에 대해 새로운 도커 이미지에 대한 sanity테스트를 추가합니다.
• 아파치 릴리스 프로세스를 확장하여 공개 DockerHub에 새로운 도커 이미지를 게시하기 위한 릴리스 스크립트를 추가합니다.

Kafka Docker 이미지에 대한 상세한 사항은 아래 링크를 참조하세요.

KIP-975: Docker Image for Apache Kafka — Apache Kafka — Apache Software Foundation

Improvement

• [KAFKA-9693] — Kafka latency spikes caused by log segment flush on roll
• [KAFKA-10199] — Separate state restoration into separate threads
• [KAFKA-14127] — KIP-858: Handle JBOD broker disk failure in KRaft
• [KAFKA-14780] — Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic
• [KAFKA-14855] — Harden integration testing logic for asserting that a connector is deleted
• [KAFKA-14912] — Introduce a configuration for remote index cache size, preferably a dynamic config.
• [KAFKA-15022] — Support rack aware task assignment in Kafka streams
• [KAFKA-15046] — Produce performance issue under high disk load
• [KAFKA-15047] — Handle rolling segments when the active segment’s retention is breached incase of tiered storage is enabled.
• [KAFKA-15141] — High CPU usage with log4j2
• [KAFKA-15147] — Measure pending and outstanding Remote Segment operations
• [KAFKA-15208] — Upgrade Jackson dependencies to version 2.16.0
• [KAFKA-15241] — Compute tiered offset by keeping the respective epochs in scope.
• [KAFKA-15248] — Add BooleanConverter to Kafka Connect
• [KAFKA-15273] — Log common name of expired client certificate
• [KAFKA-15315] — Use getOrDefault rather than get
• [KAFKA-15401] — Segment with corrupted index should not be uploaded to remote storage
• [KAFKA-15415] — In Java-client, backoff should be skipped for retried producer-batch to a new leader
• [KAFKA-15418] — Update statement on decompression location
• [KAFKA-15432] — RLM Stop partitions should not be invoked for non-tiered storage topics
• [KAFKA-15464] — Allow dynamic reloading of certificates with different DN / SANs
• [KAFKA-15471] — Allow independently stop KRaft controllers or brokers
• [KAFKA-15476] — Improve checkstyle performance
• [KAFKA-15485] — Support building with Java 21 (LTS release)
• [KAFKA-15492] — Enable spotbugs when building with Java 21
• [KAFKA-15493] — Ensure system tests work with Java 21
• [KAFKA-15521] — Refactor build.gradle to align gradle swagger plugin with swagger dependencies
• [KAFKA-15527] — Add reverseRange and reverseAll query over kv-store in IQv2
• [KAFKA-15536] — dynamically resize remoteIndexCache
• [KAFKA-15542] — Release member assignments on errors
• [KAFKA-15563] — Provide informative error messages when Connect REST requests time out
• [KAFKA-15566] — Flaky tests in FetchRequestTest.scala in KRaft mode
• [KAFKA-15596] — Upgrade ZooKeeper to 3.8.3
• [KAFKA-15629] — proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
• [KAFKA-15685] — Add missing compatibility for MinGW and MSYS2 (windows)
• [KAFKA-15769] — Fix wrong log with exception
• [KAFKA-15774] — Respect default.dsl.store Configuration Without Passing it to StreamsBuilder
• [KAFKA-15831] — List Client Metrics Configuration Resources
• [KAFKA-15837] — Throw error on use of Consumer.poll(long timeout)
• [KAFKA-15866] — Refactor OffsetFetchRequestState Error handling to be more consistent with OffsetCommitRequestState
• [KAFKA-15868] — KIP-951 — Leader discovery optimisations for the client
• [KAFKA-15906] — Emit offset syncs more often than offset.lag.max for low-throughput/finite partitions
• [KAFKA-15922] — Add MetadataVersion for JBOD
• [KAFKA-15971] — Re-enable consumer integration tests for new consumer
• [KAFKA-15980] — Add KIP-1001 CurrentControllerId metric
• [KAFKA-16007] — ZK migrations can be slow for large clusters
• [KAFKA-16210] — Upgrade jose4j to 0.9.4

Bug Fixes

• [KAFKA-12679] — Rebalancing a restoring or running task may cause directory livelocking with newly created task
• [KAFKA-13327] — Preflight validations of connectors leads to 500 responses
• [KAFKA-13973] — block-cache-capacity metrics worth twice as much as normal
• [KAFKA-13988] — Mirrormaker 2 auto.offset.reset=latest not working
• [KAFKA-14067] — Sink connector override.consumer.group.id can conflict with worker group.id
• [KAFKA-14616] — Topic recreation with offline broker causes permanent URPs
• [KAFKA-14767] — Gradle build fails with missing commitId after git gc
• [KAFKA-14927] — Prevent kafka-configs.sh from setting non-alphanumeric config key names
• [KAFKA-15000] — High vulnerability PRISMA-2023–0067 reported in jackson-core
• [KAFKA-15117] — SslTransportLayerTest.testValidEndpointIdentificationCN fails with Java 20 & 21
• [KAFKA-15152] — Fix incorrect format specifiers when formatting string
• [KAFKA-15221] — Potential race condition between requests from rebooted followers
• [KAFKA-15230] — ApiVersions data between controllers is not reliable
• [KAFKA-15311] — Fix docs about reverting to ZooKeeper mode during KRaft migration
• [KAFKA-15372] — MM2 rolling restart can drop configuration changes silently
• [KAFKA-15392] — RestServer starts but does not stop ServletContextHandler
• [KAFKA-15412] — Reading an unknown version of quorum-state-file should trigger an error
• [KAFKA-15465] — MM2 not working when its internal topics are pre-created on a cluster that disallows topic creation
• [KAFKA-15473] — Connect connector-plugins endpoint shows duplicate plugins
• [KAFKA-15481] — Concurrency bug in RemoteIndexCache leads to IOException
• [KAFKA-15489] — split brain in KRaft cluster
• [KAFKA-15491] — RackId doesn’t exist error while running WordCountDemo
• [KAFKA-15500] — Code bug in SslPrincipalMapper.java
• [KAFKA-15502] — Handle large keystores in SslEngineValidator
• [KAFKA-15507] — adminClient should not throw retriable exception when closing instance
• [KAFKA-15510] — Follower’s lastFetchedEpoch wrongly set when fetch response has no record
• [KAFKA-15511] — Exception not handled correctly if indexFile is corrupted.
• [KAFKA-15537] — Unsafe metadata.version downgrade is not supported
• [KAFKA-15571] — StateRestoreListener#onRestoreSuspended is never called because wrapper DelegatingStateRestoreListener doesn’t implement onRestoreSuspended
• [KAFKA-15602] — Breaking change in 3.4.0 ByteBufferSerializer
• [KAFKA-15605] — Topics marked for deletion in ZK are incorrectly migrated to KRaft
• [KAFKA-15607] — Possible NPE is thrown in MirrorCheckpointTask
• [KAFKA-15644] — Fix CVE-2023–4586 in netty:handler
• [KAFKA-15653] — NPE in ChunkedByteStream
• [KAFKA-15658] — Zookeeper.jar | CVE-2023–44981
• [KAFKA-15680] — Partition-Count is not getting updated Correctly in the Incremental Co-operative Rebalancing(ICR) Mode of Rebalancing
• [KAFKA-15689] — KRaftMigrationDriver not logging the skipped event when expected state is wrong
• [KAFKA-15693] — Disabling scheduled rebalance delay in Connect can lead to indefinitely unassigned connectors and tasks
• [KAFKA-15695] — Local log start offset is not updated on the follower after rebuilding remote log auxiliary state
• [KAFKA-15704] — ControllerRegistrationRequest must set ZkMigrationReady field if appropriate
• [KAFKA-15755] — LeaveGroupResponse v0-v2 should handle no members
• [KAFKA-15771] — ProduceRequest#partitionSizes() is not an atomic operation
• [KAFKA-15799] — ZK brokers incorrectly handle KRaft metadata snapshots
• [KAFKA-15800] — Malformed connect source offsets corrupt other partitions with DataException
• [KAFKA-15802] — Trying to access uncopied segments metadata on listOffsets
• [KAFKA-15817] — Avoid reconnecting to the same IP address if multiple addresses are available
• [KAFKA-15819] — KafkaServer leaks KafkaRaftManager when ZK migration enabled
• [KAFKA-15824] — SubscriptionState’s maybeValidatePositionForCurrentLeader should handle partition which isn’t subscribed yet
• [KAFKA-15825] — KRaft controller writes empty state to ZK after migration
• [KAFKA-15836] — KafkaConsumer subscribes to multiple topics does not respect max.poll.records
• [KAFKA-15860] — ControllerRegistration must be written out to the metadata image
• [KAFKA-15888] — DistributedHerder log context should not use the same client ID for each Connect worker by default
• [KAFKA-15890] — Consumer.poll with long timeout unaware of assigned partitions
• [KAFKA-15904] — Downgrade혻tests are failing with directory.id혻
• [KAFKA-15932] — Flaky test — PlaintextConsumerTest.testSeek(“kraft+kip-848”,”consumer”)
• [KAFKA-15946] — AsyncKafkaConsumer should retry commits on the application thread instead of auto-retry
• [KAFKA-15965] — Test failure: org.apache.kafka.common.requests.BrokerRegistrationRequestTest
• [KAFKA-15967] — Fix revocation in reconcilation logic
• [KAFKA-15978] — New consumer sends OffsetCommit with empty member ID
• [KAFKA-15986] — New consumer group protocol integration test failures
• [KAFKA-16005] — ZooKeeper to KRaft migration rollback missing disabling controller and migration configuration on brokers
• [KAFKA-16012] — Incomplete range assignment in consumer
• [KAFKA-16015] — kafka-leader-election timeout values always overwritten by default values
• [KAFKA-16017] — Checkpointed offset is incorrect when task is revived and restoring
• [KAFKA-16029] — Investigate cause of “Unable to find FetchSessionHandler for node X” in logs
• [KAFKA-16046] — Stream Stream Joins fail after restoration with deserialization exceptions
• [KAFKA-16078] — Be more consistent about getting the latest MetadataVersion
• [KAFKA-16085] — remote copy lag bytes/segments metrics don’t update all topic value
• [KAFKA-16094] — BrokerRegistrationRequest.logDirs field must be ignorable
• [KAFKA-16101] — KRaft migration rollback documentation is incorrect
• [KAFKA-16120] — Fix partition reassignment during ZK migration
• [KAFKA-16131] — Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2 KRaft cluster with metadata version 3.6
• [KAFKA-16133] — Commits during reconciliation always time out
• [KAFKA-16141] — StreamsStandbyTask##test_standby_tasks_rebalanceArguments:{ “metadata_quorum”: “ISOLATED_KRAFT”, “use_new_coordinator”: false} fails consistently in 3.7
• [KAFKA-16144] — Controller leader checkQuorum timer should skip only 1 controller case
• [KAFKA-16157] — Topic recreation with offline disk doesn’t update leadership/shrink ISR correctly
• [KAFKA-16162] — New created topics are unavailable after upgrading to 3.7
• [KAFKA-16216] — Reduce batch size for initial metadata load during ZK migration
• [KAFKA-16221] — IllegalStateException from Producer

더 자세한 정보는 아래 Release Notes를 참고하세요.

downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html