Kafka 2.5 → 2.6 Upgrade Guide

Kafka 2.6 brings 14 new configs, removes 0, and changes 109 defaults compared to 2.5.

+14 configs 109 defaults changed 27 KIPs

New Configurations

Config	Category	Default
ssl.engine.factory.class	broker	`null`
ssl.engine.factory.class	consumer	`null`
ssl.engine.factory.class	producer	`null`
response.http.headers.config	connect_default
ssl.engine.factory.class	connect_default	`null`
topic.creation.enable	connect_default	`true`
predicates	connect_source
topic.creation.groups	connect_source
predicates	connect_sink
acceptable.recovery.lag	stream	`10000`
default.windowed.key.serde.inner	stream	`null`
default.windowed.value.serde.inner	stream	`null`
max.warmup.replicas	stream	`2`
probing.rebalance.interval.ms	stream	`600000 (10 minutes)`

Changed Defaults

Config	Old Default	New Default
connections.max.idle.ms	`600000`	`600000 (10 minutes)`
controlled.shutdown.retry.backoff.ms	`5000`	`5000 (5 seconds)`
controller.socket.timeout.ms	`30000`	`30000 (30 seconds)`
delegation.token.expiry.check.interval.ms	`3600000`	`3600000 (1 hour)`
delegation.token.expiry.time.ms	`86400000`	`86400000 (1 day)`
delegation.token.max.lifetime.ms	`604800000`	`604800000 (7 days)`
fetch.max.bytes	`57671680`	`57671680 (55 mebibytes)`
group.initial.rebalance.delay.ms	`3000`	`3000 (3 seconds)`
group.max.session.timeout.ms	`1800000`	`1800000 (30 minutes)`
group.min.session.timeout.ms	`6000`	`6000 (6 seconds)`
inter.broker.protocol.version	`2.5-IV0`	`2.6-IV0`
log.cleaner.backoff.ms	`15000`	`15000 (15 seconds)`
log.cleaner.delete.retention.ms	`86400000`	`86400000 (1 day)`
log.flush.offset.checkpoint.interval.ms	`60000`	`60000 (1 minute)`
log.flush.start.offset.checkpoint.interval.ms	`60000`	`60000 (1 minute)`
log.index.interval.bytes	`4096`	`4096 (4 kibibytes)`
log.index.size.max.bytes	`10485760`	`10485760 (10 mebibytes)`
log.message.format.version	`2.5-IV0`	`2.6-IV0`
log.retention.check.interval.ms	`300000`	`300000 (5 minutes)`
log.segment.bytes	`1073741824`	`1073741824 (1 gibibyte)`
log.segment.delete.delay.ms	`60000`	`60000 (1 minute)`
metrics.sample.window.ms	`30000`	`30000 (30 seconds)`
offset.metadata.max.bytes	`4096`	`4096 (4 kibibytes)`
offsets.commit.timeout.ms	`5000`	`5000 (5 seconds)`
offsets.retention.check.interval.ms	`600000`	`600000 (10 minutes)`
offsets.topic.segment.bytes	`104857600`	`104857600 (100 mebibytes)`
replica.fetch.backoff.ms	`1000`	`1000 (1 second)`
replica.fetch.max.bytes	`1048576`	`1048576 (1 mebibyte)`
replica.fetch.response.max.bytes	`10485760`	`10485760 (10 mebibytes)`
replica.high.watermark.checkpoint.interval.ms	`5000`	`5000 (5 seconds)`
replica.lag.time.max.ms	`30000`	`30000 (30 seconds)`
replica.socket.receive.buffer.bytes	`65536`	`65536 (64 kibibytes)`
replica.socket.timeout.ms	`30000`	`30000 (30 seconds)`
request.timeout.ms	`30000`	`30000 (30 seconds)`
socket.receive.buffer.bytes	`102400`	`102400 (100 kibibytes)`
socket.request.max.bytes	`104857600`	`104857600 (100 mebibytes)`
socket.send.buffer.bytes	`102400`	`102400 (100 kibibytes)`
transaction.abort.timed.out.transaction.cleanup.interval.ms	`10000`	`10000 (10 seconds)`
transaction.max.timeout.ms	`900000`	`900000 (15 minutes)`
transaction.remove.expired.transaction.cleanup.interval.ms	`3600000`	`3600000 (1 hour)`
transaction.state.log.segment.bytes	`104857600`	`104857600 (100 mebibytes)`
transactional.id.expiration.ms	`604800000`	`604800000 (7 days)`
zookeeper.session.timeout.ms	`18000`	`18000 (18 seconds)`
zookeeper.sync.time.ms	`2000`	`2000 (2 seconds)`
auto.commit.interval.ms	`5000`	`5000 (5 seconds)`
client.dns.lookup	`default`	`use_all_dns_ips`
connections.max.idle.ms	`540000`	`540000 (9 minutes)`
default.api.timeout.ms	`60000`	`60000 (1 minute)`
fetch.max.bytes	`52428800`	`52428800 (50 mebibytes)`
heartbeat.interval.ms	`3000`	`3000 (3 seconds)`
max.partition.fetch.bytes	`1048576`	`1048576 (1 mebibyte)`
max.poll.interval.ms	`300000`	`300000 (5 minutes)`
metadata.max.age.ms	`300000`	`300000 (5 minutes)`
metrics.sample.window.ms	`30000`	`30000 (30 seconds)`
receive.buffer.bytes	`65536`	`65536 (64 kibibytes)`
reconnect.backoff.max.ms	`1000`	`1000 (1 second)`
request.timeout.ms	`30000`	`30000 (30 seconds)`
send.buffer.bytes	`131072`	`131072 (128 kibibytes)`
session.timeout.ms	`10000`	`10000 (10 seconds)`
client.dns.lookup	`default`	`use_all_dns_ips`
connections.max.idle.ms	`540000`	`540000 (9 minutes)`
delivery.timeout.ms	`120000`	`120000 (2 minutes)`
max.block.ms	`60000`	`60000 (1 minute)`
metadata.max.age.ms	`300000`	`300000 (5 minutes)`
metadata.max.idle.ms	`300000`	`300000 (5 minutes)`
metrics.sample.window.ms	`30000`	`30000 (30 seconds)`
receive.buffer.bytes	`32768`	`32768 (32 kibibytes)`
reconnect.backoff.max.ms	`1000`	`1000 (1 second)`
request.timeout.ms	`30000`	`30000 (30 seconds)`
send.buffer.bytes	`131072`	`131072 (128 kibibytes)`
transaction.timeout.ms	`60000`	`60000 (1 minute)`
delete.retention.ms	`86400000`	`86400000 (1 day)`
file.delete.delay.ms	`60000`	`60000 (1 minute)`
index.interval.bytes	`4096`	`4096 (4 kibibytes)`
message.format.version	`2.5-IV0`	`2.6-IV0`
retention.ms	`604800000`	`604800000 (7 days)`
segment.bytes	`1073741824`	`1073741824 (1 gibibyte)`
segment.index.bytes	`10485760`	`10485760 (10 mebibytes)`
segment.ms	`604800000`	`604800000 (7 days)`
client.dns.lookup	`default`	`use_all_dns_ips`
connections.max.idle.ms	`540000`	`540000 (9 minutes)`
heartbeat.interval.ms	`3000`	`3000 (3 seconds)`
inter.worker.key.ttl.ms	`3600000`	`3600000 (1 hour)`
metadata.max.age.ms	`300000`	`300000 (5 minutes)`
metrics.sample.window.ms	`30000`	`30000 (30 seconds)`
offset.flush.interval.ms	`60000`	`60000 (1 minute)`
offset.flush.timeout.ms	`5000`	`5000 (5 seconds)`
rebalance.timeout.ms	`60000`	`60000 (1 minute)`
receive.buffer.bytes	`32768`	`32768 (32 kibibytes)`
reconnect.backoff.max.ms	`1000`	`1000 (1 second)`
request.timeout.ms	`40000`	`40000 (40 seconds)`
scheduled.rebalance.max.delay.ms	`300000`	`300000 (5 minutes)`
send.buffer.bytes	`131072`	`131072 (128 kibibytes)`
session.timeout.ms	`10000`	`10000 (10 seconds)`
task.shutdown.graceful.timeout.ms	`5000`	`5000 (5 seconds)`
worker.sync.timeout.ms	`3000`	`3000 (3 seconds)`
worker.unsync.backoff.ms	`300000`	`300000 (5 minutes)`
errors.retry.delay.max.ms	`60000`	`60000 (1 minute)`
errors.retry.delay.max.ms	`60000`	`60000 (1 minute)`
commit.interval.ms	`30000`	`30000 (30 seconds)`
connections.max.idle.ms	`540000`	`540000 (9 minutes)`
metadata.max.age.ms	`300000`	`300000 (5 minutes)`
metrics.sample.window.ms	`30000`	`30000 (30 seconds)`
receive.buffer.bytes	`32768`	`32768 (32 kibibytes)`
reconnect.backoff.max.ms	`1000`	`1000 (1 second)`
request.timeout.ms	`40000`	`40000 (40 seconds)`
send.buffer.bytes	`131072`	`131072 (128 kibibytes)`
state.cleanup.delay.ms	`600000`	`600000 (10 minutes)`
windowstore.changelog.additional.retention.ms	`86400000`	`86400000 (1 day)`

Protocol Changes

ListGroups v4 · ListGroups v4 · DeleteRecords v2 · DeleteRecords v2 · DescribeConfigs v3 · DescribeConfigs v3 · DescribeLogDirs v2 · DescribeLogDirs v2

KIPs Released

KIP-131 Add access to OffsetStorageReader from SourceConnector
KIP-158 Kafka Connect should allow source connectors to set topic-specific settings for new topics
KIP-221 Enhance DSL with Connecting Topic Creation and Repartition Hint
KIP-422 KIP-421
KIP-437 Custom replacement for MaskField SMT
KIP-518 Allow listing consumer groups per state
KIP-535 Allow state stores to serve stale reads during rebalance
KIP-544 Make metrics exposed via JMX configurable
KIP-546 Add Client Quota APIs to the Admin Client
KIP-551 Expose disk read and write metrics
KIP-557 Add emit on change support for Kafka Streams
KIP-568 Explicit rebalance triggering on the Consumer
KIP-569 DescribeConfigsResponse - Update the schema to include additional metadata information of the field
KIP-570 Add leader epoch in StopReplicaRequest
KIP-571 Add option to force remove members in StreamsResetter
KIP-573 Enable TLSv1.3 by default
KIP-574 CLI Dynamic Configuration with file input
KIP-585 Filter and Conditional SMTs
KIP-586 Deprecate commit records without record metadata
KIP-594 Expose output topic names from TopologyTestDriver
KIP-602 Change default value for client.dns.lookup
KIP-604 Remove ZooKeeper Flags from the Administrative Tools
KIP-605 Expand Connect Worker Internal Topic Settings
KIP-606 Add Metadata Context to MetricsReporter
KIP-610 Error Reporting in Sink Connectors
KIP-613 Add end-to-end latency metrics to Streams
KIP-1137 Standardize Configuration Precedence for Tool Scripts

Planning a Kafka upgrade? Conduktor Console helps you manage and monitor your clusters during migration.