KIP-283 — Efficient Memory Usage for Down-Conversion

Accepted Kafka 2.0 BrokerProtocol

Implements lazy, chunked down-conversion for older fetch protocol clients, converting only the requested bytes at a time rather than materializing the entire partition's unconverted data in heap memory. When brokers must down-convert messages to an older format for legacy consumers, they previously loaded entire log segments into JVM heap, causing OutOfMemoryErrors under concurrent down-conversion load.

Protocol Impact

Fetch

Details