Kafka 3.7 → 3.8 Upgrade Guide

Kafka 3.8 brings 26 new configs, removes 0, and changes 4 defaults compared to 3.7.

+26 configs 4 defaults changed 20 KIPs

New Configurations

Config	Category	Default
compression.gzip.level	broker	-1
compression.lz4.level	broker	9
compression.zstd.level	broker	3
group.consumer.migration.policy	broker	disabled
group.coordinator.append.linger.ms	broker	10
log.dir.failure.timeout.ms	broker	30000 (30 seconds)
max.request.partition.size.limit	broker	2000
remote.fetch.max.wait.ms	broker	500
remote.log.manager.copy.max.bytes.per.second	broker	9223372036854775807
remote.log.manager.copy.quota.window.num	broker	11
remote.log.manager.copy.quota.window.size.seconds	broker	1
remote.log.manager.fetch.max.bytes.per.second	broker	9223372036854775807
remote.log.manager.fetch.quota.window.num	broker	11
remote.log.manager.fetch.quota.window.size.seconds	broker	1
metadata.recovery.strategy	consumer	none
compression.gzip.level	producer	-1
compression.lz4.level	producer	9
compression.zstd.level	producer	3
metadata.recovery.strategy	producer	none
compression.gzip.level	topic	-1
compression.lz4.level	topic	9
compression.zstd.level	topic	3
metadata.recovery.strategy	connect_default	none
tasks.max.enforce	connect_source	true
tasks.max.enforce	connect_sink	true
task.assignor.class	stream	null

Changed Defaults

Config	Old Default	New Default
inter.broker.protocol.version	3.7-IV4	3.8-IV0
listener.security.protocol.map	PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL	SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT
client.id		<application.id>-<random-UUID>
state.dir	/var/folders/tj/qjgd_zb13w19nkmt8rfpnssc0000gn/T//kafka-streams	${java.io.tmpdir}

Protocol Changes

Produce v11 · Produce v11 · FindCoordinator v5 · FindCoordinator v5 · ListGroups v5 · ListGroups v5 · InitProducerId v5 · InitProducerId v5 · AddPartitionsToTxn v5 · AddPartitionsToTxn v5 · AddOffsetsToTxn v4 · AddOffsetsToTxn v4 · EndTxn v4 · EndTxn v4 · TxnOffsetCommit v4 · TxnOffsetCommit v4 · ListTransactions v1 · ListTransactions v1

KIPs Released

KIP-390 Support Compression Level
KIP-477 Add PATCH method for connector config in Connect REST API
KIP-719 Deprecate Log4J Appender
KIP-813 Shareable State Stores
KIP-862 Self-join optimization for stream-stream joins
KIP-869 Improve Streams State Restoration Visibility
KIP-890 Transactions Server-Side Defense
KIP-899 Allow clients to rebootstrap
KIP-924 customizable task assignment for Streams
KIP-945 Update threading model for Consumer
KIP-974 Docker Image for GraalVM based Native Kafka Broker
KIP-989 Improved StateStore Iterator metrics for detecting leaks
KIP-993 Allow restricting files accessed by File and Directory ConfigProviders
KIP-994 Minor Enhancements to ListTransactions and DescribeTransactions APIs
KIP-1004 Enforce tasks.max property in Kafka Connect
KIP-1018 Introduce max remote fetch timeout config for DelayedRemoteFetch requests
KIP-1019 Expose method to determine Metric Measurability
KIP-1028 Docker Official Image for Apache Kafka
KIP-1036 Extend RecordDeserializationException exception
KIP-1037 Allow WriteTxnMarkers API with Alter Cluster Permission