Kafka 3.9 → 4.0 Upgrade Guide

Kafka 4.0 brings 23 new configs, removes 49, and changes 28 defaults compared to 3.9.

+23 configs −49 configs 28 defaults changed 31 KIPs

Removed Configurations

Config	Category
alter.config.policy.kraft.compatibility.enable	broker
auto.include.jmx.reporter	broker
broker.id.generation.enable	broker
control.plane.listener.name	broker
controlled.shutdown.max.retries	broker
controlled.shutdown.retry.backoff.ms	broker
delegation.token.master.key	broker
eligible.leader.replicas.enable	broker
inter.broker.protocol.version	broker
leader.imbalance.per.broker.percentage	broker
log.message.downconversion.enable	broker
log.message.format.version	broker
log.message.timestamp.difference.max.ms	broker
offsets.commit.required.acks	broker
password.encoder.cipher.algorithm	broker
password.encoder.iterations	broker
password.encoder.key.length	broker
password.encoder.keyfactory.algorithm	broker
password.encoder.old.secret	broker
password.encoder.secret	broker
reserved.broker.max.id	broker
zookeeper.clientCnxnSocket	broker
zookeeper.connect	broker
zookeeper.connection.timeout.ms	broker
zookeeper.max.in.flight.requests	broker
zookeeper.metadata.migration.enable	broker
zookeeper.session.timeout.ms	broker
zookeeper.set.acl	broker
zookeeper.ssl.cipher.suites	broker
zookeeper.ssl.client.enable	broker
zookeeper.ssl.crl.enable	broker
zookeeper.ssl.enabled.protocols	broker
zookeeper.ssl.endpoint.identification.algorithm	broker
zookeeper.ssl.keystore.location	broker
zookeeper.ssl.keystore.password	broker
zookeeper.ssl.keystore.type	broker
zookeeper.ssl.ocsp.enable	broker
zookeeper.ssl.protocol	broker
zookeeper.ssl.truststore.location	broker
zookeeper.ssl.truststore.password	broker
zookeeper.ssl.truststore.type	broker
auto.include.jmx.reporter	consumer
auto.include.jmx.reporter	producer
message.downconversion.enable	topic
message.format.version	topic
message.timestamp.difference.max.ms	topic
auto.include.jmx.reporter	connect_default
auto.include.jmx.reporter	stream
retries	stream

New Configurations

Config	Category	Default
add.partitions.to.txn.retry.backoff.max.ms	broker	100
add.partitions.to.txn.retry.backoff.ms	broker	20
group.consumer.migration.policy	broker	bidirectional
remote.list.offsets.request.timeout.ms	broker	30000 (30 seconds)
share.coordinator.append.linger.ms	broker	10
share.coordinator.load.buffer.size	broker	5242880
share.coordinator.snapshot.update.records.per.snapshot	broker	500
share.coordinator.state.topic.compression.codec	broker	0
share.coordinator.state.topic.min.isr	broker	2
share.coordinator.state.topic.num.partitions	broker	50
share.coordinator.state.topic.replication.factor	broker	3
share.coordinator.state.topic.segment.bytes	broker	104857600 (100 mebibytes)
share.coordinator.threads	broker	1
share.coordinator.write.timeout.ms	broker	5000 (5 seconds)
share.fetch.purgatory.purge.interval.requests	broker	1000
metadata.recovery.rebootstrap.trigger.ms	consumer	300000 (5 minutes)
metadata.recovery.rebootstrap.trigger.ms	producer	300000 (5 minutes)
metadata.recovery.rebootstrap.trigger.ms	connect_default	300000 (5 minutes)
deserialization.exception.handler	stream	org.apache.kafka.streams.errors.LogAndFailExceptionHandler
metadata.recovery.rebootstrap.trigger.ms	stream	300000 (5 minutes)
metadata.recovery.strategy	stream	rebootstrap
processor.wrapper.class	stream	org.apache.kafka.streams.processor.internals.NoOpProcessorWrapper
production.exception.handler	stream	org.apache.kafka.streams.errors.DefaultProductionExceptionHandler

Changed Defaults

Config	Old Default	New Default
group.consumer.assignors	org.apache.kafka.coordinator.group.assignor.UniformAssignor,org.apache.kafka.coordinator.group.assignor.RangeAssignor	uniform,range
group.coordinator.append.linger.ms	10	5
group.coordinator.rebalance.protocols	classic	classic,consumer
group.coordinator.threads	1	4
log.message.timestamp.after.max.ms	9223372036854775807	3600000 (1 hour)
metric.reporters		org.apache.kafka.common.metrics.JmxReporter
node.id	-1
num.recovery.threads.per.data.dir	1	2
remote.log.manager.copier.thread.pool.size	-1	10
remote.log.manager.expiration.thread.pool.size	-1	10
remote.log.manager.thread.pool.size	10	2
ssl.enabled.protocols	TLSv1.2	TLSv1.2,TLSv1.3
ssl.protocol	TLSv1.2	TLSv1.3
metadata.recovery.strategy	none	rebootstrap
metric.reporters		org.apache.kafka.common.metrics.JmxReporter
ssl.enabled.protocols	TLSv1.2	TLSv1.2,TLSv1.3
ssl.protocol	TLSv1.2	TLSv1.3
linger.ms	0	5
metadata.recovery.strategy	none	rebootstrap
metric.reporters		org.apache.kafka.common.metrics.JmxReporter
ssl.enabled.protocols	TLSv1.2	TLSv1.2,TLSv1.3
ssl.protocol	TLSv1.2	TLSv1.3
message.timestamp.after.max.ms	9223372036854775807	3600000 (1 hour)
metadata.recovery.strategy	none	rebootstrap
metric.reporters		org.apache.kafka.common.metrics.JmxReporter
ssl.enabled.protocols	TLSv1.2	TLSv1.2,TLSv1.3
ssl.protocol	TLSv1.2	TLSv1.3
metric.reporters		org.apache.kafka.common.metrics.JmxReporter

Protocol Changes

Produce v12 · Produce v12 · ListOffsets v10 · ListOffsets v10 · Metadata v13 · Metadata v13 · DescribeGroups v6 · DescribeGroups v6 · EndTxn v5 · EndTxn v5 · TxnOffsetCommit v5 · TxnOffsetCommit v5 · Vote v2 · Vote v2 · UpdateFeatures v2 · UpdateFeatures v2 · DescribeCluster v2 · DescribeCluster v2 · ConsumerGroupHeartbeat v1 · ConsumerGroupHeartbeat v1 · ConsumerGroupDescribe v1 · ConsumerGroupDescribe v1

KIPs Released

KIP-653 Upgrade log4j to log4j2
KIP-891 Running multiple versions of Connector plugins
KIP-970 Deprecate and remove Connect's redundant task configurations endpoint
KIP-996 Pre-Vote
KIP-1011 Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh
KIP-1022 Formatting and Updating Features
KIP-1030 Change constraints and default values for various configurations
KIP-1032 Upgrade to Jakarta and JavaEE 10 in Kafka 4.0
KIP-1043 Administration of groups
KIP-1056 Remove `default.` prefix for exception handler StreamsConfig
KIP-1058 Txn consumer exerts pressure on remote storage when reading non-txn topic
KIP-1068 New metrics for the new KafkaConsumer
KIP-1070 deprecate MockProcessorContext
KIP-1073 Return fenced brokers in DescribeCluster response
KIP-1074 Allow the replication of user internal topics
KIP-1075 Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async
KIP-1076 Metrics for client applications KIP-714 extension
KIP-1077 Deprecate `ForeachProcessor` and move to internal package
KIP-1078 Remove Leaking Getter Methods in Joined Helper Class
KIP-1079 Deprecate `delete-config` of TopicCommand
KIP-1080 Fix the typo: `maxlifeTimeMs` in CreateDelegationTokenOptions
KIP-1082 Require Client-Generated IDs over the ConsumerGroupHeartbeat RPC
KIP-1089 Allow disabling heartbeats replication in MirrorSourceConnector
KIP-1091 Improved Kafka Streams operator metrics
KIP-1094 Add a new constructor with nextOffsets to ConsumerRecords
KIP-1099 Extend kafka-consumer-groups command line tool to support new consumer group
KIP-1101 Trigger rebalance on rack topology changes
KIP-1102 Enable clients to rebootstrap based on timeout or error code
KIP-1105 Make remote log manager thread-pool configs dynamic
KIP-1106 Add duration based offset reset option for consumer clients
KIP-1112 allow custom processor wrapping