Kafka 3.5 → 3.6 Upgrade Guide

Kafka 3.6 brings 27 new configs, removes 0, and changes 2 defaults compared to 3.5.

+27 configs 2 defaults changed 13 KIPs

New Configurations

Config	Category	Default
log.local.retention.bytes	broker	-2
log.local.retention.ms	broker	-2
log.message.timestamp.after.max.ms	broker	9223372036854775807
log.message.timestamp.before.max.ms	broker	9223372036854775807
remote.log.manager.task.interval.ms	broker	30000 (30 seconds)
remote.log.manager.thread.pool.size	broker	10
remote.log.metadata.custom.metadata.max.bytes	broker	128
remote.log.metadata.manager.class.name	broker	org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.class.path	broker	null
remote.log.metadata.manager.impl.prefix	broker	rlmm.config.
remote.log.metadata.manager.listener.name	broker	null
remote.log.reader.max.pending.tasks	broker	100
remote.log.reader.threads	broker	10
remote.log.storage.manager.class.name	broker	null
remote.log.storage.manager.class.path	broker	null
remote.log.storage.manager.impl.prefix	broker	rsm.config.
remote.log.storage.system.enable	broker	false
transaction.partition.verification.enable	broker	true
local.retention.bytes	topic	-2
local.retention.ms	topic	-2
message.timestamp.after.max.ms	topic	9223372036854775807
message.timestamp.before.max.ms	topic	9223372036854775807
remote.storage.enable	topic	false
plugin.discovery	connect_default	hybrid_warn
rack.aware.assignment.non_overlap_cost	stream	null
rack.aware.assignment.strategy	stream	none
rack.aware.assignment.traffic_cost	stream	null

Changed Defaults

Config	Old Default	New Default
inter.broker.protocol.version	3.5-IV2	3.6-IV2
state.dir	/var/folders/qq/2qmvd8cd11x3fcd6wbgpn9pw0000gn/T//kafka-streams	/var/folders/3j/8r9d0znd5pzgp8ww95yn8g140000gp/T//kafka-streams

Protocol Changes

OffsetCommit v9 · OffsetCommit v9

KIPs Released

KIP-495 Dynamically Adjust Log Levels in Connect
KIP-793 Allow sink connectors to be used with topic-mutating SMTs
KIP-848 The Next Generation of the Consumer Rebalance Protocol
KIP-852 Optimize calculation of size for log in remote tier
KIP-868 Metadata Transactions
KIP-898 Modernize Connect plugin discovery
KIP-902 Upgrade Zookeeper to 3.8.2
KIP-917 Additional custom metadata for remote log segment
KIP-925 Rack aware task assignment in Kafka Streams
KIP-927 Improve the kafka-metadata-quorum output
KIP-930 Rename ambiguous Tiered Storage Metrics
KIP-937 Improve Message Timestamp Validation
KIP-949 Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy