Kafka 3.0 → 3.1 Upgrade Guide

Kafka 3.1 brings 56 new configs, removes 3, and changes 10 defaults compared to 3.0.

+56 configs −3 configs 10 defaults changed 8 KIPs

Removed Configurations

Config	Category
local.retention.bytes	topic
local.retention.ms	topic
remote.storage.enable	topic

New Configurations

Config	Category	Default
sasl.login.connect.timeout.ms	broker	null
sasl.login.read.timeout.ms	broker	null
sasl.login.retry.backoff.max.ms	broker	10000 (10 seconds)
sasl.login.retry.backoff.ms	broker	100
sasl.oauthbearer.clock.skew.seconds	broker	30
sasl.oauthbearer.expected.audience	broker	null
sasl.oauthbearer.expected.issuer	broker	null
sasl.oauthbearer.jwks.endpoint.refresh.ms	broker	3600000 (1 hour)
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms	broker	10000 (10 seconds)
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms	broker	100
sasl.oauthbearer.jwks.endpoint.url	broker	null
sasl.oauthbearer.scope.claim.name	broker	scope
sasl.oauthbearer.sub.claim.name	broker	sub
sasl.oauthbearer.token.endpoint.url	broker	null
sasl.login.connect.timeout.ms	consumer	null
sasl.login.read.timeout.ms	consumer	null
sasl.login.retry.backoff.max.ms	consumer	10000 (10 seconds)
sasl.login.retry.backoff.ms	consumer	100
sasl.oauthbearer.clock.skew.seconds	consumer	30
sasl.oauthbearer.expected.audience	consumer	null
sasl.oauthbearer.expected.issuer	consumer	null
sasl.oauthbearer.jwks.endpoint.refresh.ms	consumer	3600000 (1 hour)
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms	consumer	10000 (10 seconds)
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms	consumer	100
sasl.oauthbearer.jwks.endpoint.url	consumer	null
sasl.oauthbearer.scope.claim.name	consumer	scope
sasl.oauthbearer.sub.claim.name	consumer	sub
sasl.oauthbearer.token.endpoint.url	consumer	null
sasl.login.connect.timeout.ms	producer	null
sasl.login.read.timeout.ms	producer	null
sasl.login.retry.backoff.max.ms	producer	10000 (10 seconds)
sasl.login.retry.backoff.ms	producer	100
sasl.oauthbearer.clock.skew.seconds	producer	30
sasl.oauthbearer.expected.audience	producer	null
sasl.oauthbearer.expected.issuer	producer	null
sasl.oauthbearer.jwks.endpoint.refresh.ms	producer	3600000 (1 hour)
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms	producer	10000 (10 seconds)
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms	producer	100
sasl.oauthbearer.jwks.endpoint.url	producer	null
sasl.oauthbearer.scope.claim.name	producer	scope
sasl.oauthbearer.sub.claim.name	producer	sub
sasl.oauthbearer.token.endpoint.url	producer	null
sasl.login.connect.timeout.ms	connect_default	null
sasl.login.read.timeout.ms	connect_default	null
sasl.login.retry.backoff.max.ms	connect_default	10000 (10 seconds)
sasl.login.retry.backoff.ms	connect_default	100
sasl.oauthbearer.clock.skew.seconds	connect_default	30
sasl.oauthbearer.expected.audience	connect_default	null
sasl.oauthbearer.expected.issuer	connect_default	null
sasl.oauthbearer.jwks.endpoint.refresh.ms	connect_default	3600000 (1 hour)
... and 6 more

Changed Defaults

Config	Old Default	New Default
inter.broker.protocol.version	3.0-IV1	3.1-IV0
ssl.enabled.protocols	TLSv1.2	TLSv1.2,TLSv1.3
ssl.protocol	TLSv1.2	TLSv1.3
ssl.enabled.protocols	TLSv1.2	TLSv1.2,TLSv1.3
ssl.protocol	TLSv1.2	TLSv1.3
ssl.enabled.protocols	TLSv1.2	TLSv1.2,TLSv1.3
ssl.protocol	TLSv1.2	TLSv1.3
ssl.enabled.protocols	TLSv1.2	TLSv1.2,TLSv1.3
ssl.protocol	TLSv1.2	TLSv1.3
state.dir	/var/folders/5w/m48dfpps5fj1byw1ldmq3v5w0000gp/T//kafka-streams	/tmp/kafka-streams

Protocol Changes

Fetch v13 · Fetch v13 · Metadata v12 · Metadata v12

KIPs Released

KIP-534 Reorganize checkpoint system in log cleaner to per partition
KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention
KIP-748 Add Broker Count Metrics
KIP-761 Add Total Blocked Time Metric to Streams
KIP-763 Range Queries with Open Endpoints
KIP-773 Differentiate consistently metric latency measured in millis and nanos
KIP-775 Custom partitioners in foreign key joins
KIP-783 Add TaskId field to StreamsException