Kafka 2.3 → 2.4 Upgrade Guide

Kafka 2.4 brings 10 new configs, removes 0, and changes 23 defaults compared to 2.3.

+10 configs 23 defaults changed 34 KIPs

New Configurations

Config	Category	Default
replica.selector.class	broker	null
security.providers	broker	null
security.providers	consumer	null
security.providers	producer	null
admin.listeners	connect_default	null
inter.worker.key.generation.algorithm	connect_default	HmacSHA256
inter.worker.key.size	connect_default	null
inter.worker.key.ttl.ms	connect_default	3600000
inter.worker.signature.algorithm	connect_default	HmacSHA256
inter.worker.verification.algorithms	connect_default	HmacSHA256

Changed Defaults

Config	Old Default	New Default
inter.broker.protocol.version	2.3-IV1	2.4-IV1
listener.security.protocol.map	PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL	PLAINTEXT
log.message.format.version	2.3-IV1	2.4-IV1
zookeeper.connect	null
key.deserializer	null
value.deserializer	null
key.serializer	null
value.serializer	null
message.format.version	2.3-IV1	2.4-IV1
bootstrap.servers	localhost:9092	localhost
config.storage.topic	null
connect.protocol	compatible	sessioned
group.id	null
key.converter	null
offset.storage.topic	null
status.storage.topic	null
value.converter	null
connector.class	null
name	null
connector.class	null
name	null
application.id	null
bootstrap.servers	null

Protocol Changes

Produce v8 · Produce v8 · Metadata v9 · Metadata v9 · OffsetCommit v8 · OffsetCommit v8 · OffsetFetch v6 · OffsetFetch v6 · FindCoordinator v3 · FindCoordinator v3 · JoinGroup v6 · JoinGroup v6 · Heartbeat v4 · Heartbeat v4 · LeaveGroup v3 · LeaveGroup v3 · SyncGroup v4 · SyncGroup v4 · DescribeGroups v4 · DescribeGroups v4 · ListGroups v3 · ListGroups v3 · ApiVersions v3 · ApiVersions v3 · CreateTopics v4 · CreateTopics v4 · DeleteTopics v4 · DeleteTopics v4 · InitProducerId v2 · InitProducerId v2 · CreateDelegationToken v2 · CreateDelegationToken v2 · DeleteGroups v2 · DeleteGroups v2 · ElectLeaders v1 · ElectLeaders v1 · IncrementalAlterConfigs v1 · IncrementalAlterConfigs v1

KIPs Released

KIP-213 Support non-key joining in KTable
KIP-309 Add toUpperCase support to sasl.kerberos.principal.to.local rule
KIP-379 Multiple Consumer Group Management
KIP-382 MirrorMaker 2.0
KIP-412 Extend Admin API to support dynamic application log levels
KIP-416 Notify SourceTask of ACK'd offsets, metadata
KIP-429 Kafka Consumer Incremental Rebalance Protocol
KIP-434 Add Replica Fetcher and Log Cleaner Count Metrics
KIP-440 Extend Connect Converter to support headers
KIP-444 Augment metrics for Kafka Streams
KIP-455  Create an Administrative API for Replica Reassignment
KIP-460 Admin Leader Election RPC
KIP-467 Augment ProduceResponse error messaging for specific culprit records
KIP-470 TopologyTestDriver test input and output usability improvements
KIP-471 Expose RocksDB Metrics in Kafka Streams
KIP-475 New Metrics to Measure Number of Tasks on a Connector
KIP-476 Add Java AdminClient Interface
KIP-479 Add StreamJoined config object to Join
KIP-480 Sticky Partitioner
KIP-481 SerDe Improvements for Connect Decimal type in JSON
KIP-482 The Kafka Protocol should Support Optional Tagged Fields
KIP-484 Expose metrics for group and transaction metadata loading duration
KIP-488 Clean up Sum,Count,Total Metrics
KIP-492 Add java security providers in Kafka Security config
KIP-496 Administrative API to delete consumer offsets
KIP-503 Add metric for number of topics marked for deletion
KIP-504 Add new Java Authorizer Interface
KIP-507  Securing Internal Connect REST Endpoints
KIP-511 Collect and Expose Client's Name and Version in the Brokers
KIP-517 Add consumer metrics to observe user poll behavior
KIP-521 Enable redirection of Connect's log4j messages to a file by default
KIP-525 Return topic metadata and configs in CreateTopics response
KIP-528 Deprecate PartitionGrouper configuration and interface
KIP-559 Make the Kafka Protocol Friendlier with L7 Proxies