KIP-31 — Move to relative offsets in compressed message sets

Accepted Kafka 1.0 BrokerProtocol

Changes the message format in compressed message sets to use relative (inner) offsets instead of absolute offsets, so the broker can append the message set without decompressing and re-compressing it just to assign offsets. The broker was forced to decompress every compressed batch, assign per-message absolute offsets, and recompress — a significant CPU cost on the hot write path.

Protocol Impact

Produce · Fetch

Details