Kafka 4.1 → 4.2 Upgrade Guide

Kafka 4.2 brings 10 new configs, removes 0, and changes 11 defaults compared to 4.1.

+10 configs 11 defaults changed 34 KIPs

New Configurations

Config	Category	Default
config.providers	broker
controller.quorum.auto.join.enable	broker	false
group.streams.initial.rebalance.delay.ms	broker	3000 (3 seconds)
remote.log.manager.follower.thread.pool.size	broker	2
config.providers	consumer
share.acquire.mode	consumer	BATCH_OPTIMIZED
config.providers	producer
allow.os.group.write.access	stream	false
config.providers	stream
errors.dead.letter.queue.topic.name	stream	null

Changed Defaults

Config	Old Default	New Default
controller.listener.names	null
group.coordinator.append.linger.ms	5	-1
sasl.oauthbearer.expected.audience	null
share.coordinator.append.linger.ms	5	-1
sasl.oauthbearer.expected.audience	null
ssl.cipher.suites	null
sasl.oauthbearer.expected.audience	null
ssl.cipher.suites	null
bootstrap.servers	localhost:9092
sasl.oauthbearer.expected.audience	null
ssl.cipher.suites	null

Protocol Changes

ListOffsets v11 · ListOffsets v11 · WriteTxnMarkers v2 · WriteTxnMarkers v2 · ShareFetch v2 · ShareFetch v2 · ShareAcknowledge v2 · ShareAcknowledge v2 · AddRaftVoter v1 · AddRaftVoter v1 · WriteShareGroupState v1 · WriteShareGroupState v1 · ReadShareGroupStateSummary v1 · ReadShareGroupStateSummary v1 · DescribeShareGroupOffsets v1 · DescribeShareGroupOffsets v1

KIPs Released

KIP-711 Deprecate org.apache.kafka.streams.errors.BrokerNotFoundException
KIP-1023 Follower fetch from tiered offset
KIP-1034 Dead letter queue in Kafka Streams
KIP-1052 Enable warmup in producer performance test
KIP-1054 Support External schema in JSONConvertor
KIP-1100 Rename org.apache.kafka.server:type=AssignmentsManager and org.apache.kafka.storage.internals.log.RemoteStorageThreadPool metrics
KIP-1120 AppInfo metrics don't contain the client-id
KIP-1136 Make ConsumerGroupMetadata an interface
KIP-1146 Anchored punctuation
KIP-1147 Improve consistency of command-line arguments
KIP-1153 Refactor Kafka Streams CloseOptions to Fluent API Style
KIP-1157 Enforce KafkaPrincipalSerde Implementation for KafkaPrincipalBuilder
KIP-1160 Enable returning supported features from a specific broker
KIP-1161 Unifying LIST-Type Configuration Validation and Default Values
KIP-1175 Fix the typo `PARTITIONER_ADPATIVE_PARTITIONING_ENABLE` in ProducerConfig
KIP-1180 Add generic feature level metrics
KIP-1186 Update AddRaftVoterRequest RPC to support auto-join
KIP-1188 New ConnectorClientConfigOverridePolicy with allowlist of configurations
KIP-1190 Add a metric for controller thread idleness
KIP-1192 Add include argument to ConsumerPerformance tool
KIP-1193 Deprecate MX4j support
KIP-1195 deprecate and remove org.apache.kafka.streams.errors.BrokerNotFoundException
KIP-1197 Introduce new method to improve the TopicBasedRemoteLogMetadataManager's initialization
KIP-1206 Strict max fetch records in share fetch
KIP-1207 Fix anomaly of JMX metrics RequestHandlerAvgIdlePercent in kraft combined mode
KIP-1216 Add rebalance listener metrics for Kafka Streams
KIP-1217 Include push interval in ClientTelemetryReceiver context
KIP-1221 Add application-id tag to Kafka Streams state metric
KIP-1222 Acquisition lock timeout renewal in share consumer explicit mode
KIP-1224 Adaptive append.linger.ms for the group coordinator and share coordinator
KIP-1226 Introducing Share Partition Lag Persistence and Retrieval
KIP-1227 Expose Rack ID in MemberDescription and ShareMemberDescription
KIP-1228 Add Transaction Version to WriteTxnMarkersRequest
KIP-1230 Add config for file system permission