Kafka 4.0 → 4.1 Upgrade Guide

Kafka 4.1 brings 91 new configs, removes 1, and changes 3 defaults compared to 4.0.

+91 configs −1 configs 3 defaults changed 19 KIPs

Removed Configurations

Config	Category
group.share.max.groups	broker

New Configurations

Config	Category	Default
group.share.assignors	broker	simple
group.share.max.share.sessions	broker	2000
group.streams.heartbeat.interval.ms	broker	5000 (5 seconds)
group.streams.max.heartbeat.interval.ms	broker	15000 (15 seconds)
group.streams.max.session.timeout.ms	broker	60000 (1 minute)
group.streams.max.size	broker	2147483647
group.streams.max.standby.replicas	broker	2
group.streams.min.heartbeat.interval.ms	broker	5000 (5 seconds)
group.streams.min.session.timeout.ms	broker	45000 (45 seconds)
group.streams.num.standby.replicas	broker	0
group.streams.session.timeout.ms	broker	45000 (45 seconds)
sasl.oauthbearer.assertion.algorithm	broker	RS256
sasl.oauthbearer.assertion.claim.aud	broker	null
sasl.oauthbearer.assertion.claim.exp.seconds	broker	300
sasl.oauthbearer.assertion.claim.iss	broker	null
sasl.oauthbearer.assertion.claim.jti.include	broker	false
sasl.oauthbearer.assertion.claim.nbf.seconds	broker	60
sasl.oauthbearer.assertion.claim.sub	broker	null
sasl.oauthbearer.assertion.file	broker	null
sasl.oauthbearer.assertion.private.key.file	broker	null
sasl.oauthbearer.assertion.private.key.passphrase	broker	null
sasl.oauthbearer.assertion.template.file	broker	null
sasl.oauthbearer.client.credentials.client.id	broker	null
sasl.oauthbearer.client.credentials.client.secret	broker	null
sasl.oauthbearer.jwt.retriever.class	broker	org.apache.kafka.common.security.oauthbearer.DefaultJwtRetriever
sasl.oauthbearer.jwt.validator.class	broker	org.apache.kafka.common.security.oauthbearer.DefaultJwtValidator
sasl.oauthbearer.scope	broker	null
transaction.two.phase.commit.enable	broker	false
sasl.oauthbearer.assertion.algorithm	consumer	RS256
sasl.oauthbearer.assertion.claim.aud	consumer	null
sasl.oauthbearer.assertion.claim.exp.seconds	consumer	300
sasl.oauthbearer.assertion.claim.iss	consumer	null
sasl.oauthbearer.assertion.claim.jti.include	consumer	false
sasl.oauthbearer.assertion.claim.nbf.seconds	consumer	60
sasl.oauthbearer.assertion.claim.sub	consumer	null
sasl.oauthbearer.assertion.file	consumer	null
sasl.oauthbearer.assertion.private.key.file	consumer	null
sasl.oauthbearer.assertion.private.key.passphrase	consumer	null
sasl.oauthbearer.assertion.template.file	consumer	null
sasl.oauthbearer.client.credentials.client.id	consumer	null
sasl.oauthbearer.client.credentials.client.secret	consumer	null
sasl.oauthbearer.jwt.retriever.class	consumer	org.apache.kafka.common.security.oauthbearer.DefaultJwtRetriever
sasl.oauthbearer.jwt.validator.class	consumer	org.apache.kafka.common.security.oauthbearer.DefaultJwtValidator
sasl.oauthbearer.scope	consumer	null
share.acknowledgement.mode	consumer	implicit
sasl.oauthbearer.assertion.algorithm	producer	RS256
sasl.oauthbearer.assertion.claim.aud	producer	null
sasl.oauthbearer.assertion.claim.exp.seconds	producer	300
sasl.oauthbearer.assertion.claim.iss	producer	null
sasl.oauthbearer.assertion.claim.jti.include	producer	false
... and 41 more

Changed Defaults

Config	Old Default	New Default
group.coordinator.rebalance.protocols	classic,consumer	classic,consumer,streams
group.share.partition.max.record.locks	200	2000
share.coordinator.append.linger.ms	10	5

Protocol Changes

Produce v13 · Produce v13 · Fetch v18 · Fetch v18 · OffsetCommit v10 · OffsetCommit v10 · OffsetFetch v10 · OffsetFetch v10 · InitProducerId v6 · InitProducerId v6 · AlterPartitionReassignments v1 · AlterPartitionReassignments v1 · ListTransactions v2 · ListTransactions v2 · ListConfigResources v1 · ListConfigResources v1

KIPs Released

KIP-512 make Record Headers available in onAcknowledgement and onComplete
KIP-516 Topic Identifiers & Topic Deletion State Improvements
KIP-860 Add client-provided option to guard against replication factor change during partition reassignments
KIP-877 Mechanism for plugins and connectors to register metrics
KIP-939 Support Participation in 2PC
KIP-1050 Consistent error handling for Transactions
KIP-1071 Streams Rebalance Protocol
KIP-1103 Additional metrics for cooperative consumption
KIP-1104 Allow Foreign Key Extraction from Both Key and Value in KTable Joins
KIP-1109 Unifying Kafka Consumer Topic Metrics
KIP-1111 Enforcing Explicit Naming for Kafka Streams Internal Topics
KIP-1118 Add Deadlock Protection on Producer Network Thread
KIP-1131 Improved controller-side monitoring of broker states
KIP-1132 KRaft servers support invalid static SocketServer configurations
KIP-1139 Add support for OAuth jwt-bearer grant type
KIP-1142 Allow to list non-existent group which has dynamic config
KIP-1148 Remove log.cleaner.enable and set lower bound 1 to log.cleaner.threads
KIP-1152 Add transactional ID pattern filter to ListTransactions API
KIP-1166 Improve high-watermark replication