KIP-1332 — Dynamic memory allocation for the Kafka producer

Accepted Producer

Replaces the producer's fixed per-partition allocation, which reserves a full batch.size of pool memory for every partition that starts a batch, with dynamic allocation sized to the data actually buffered. Pool reservation scaling with active-partition count rather than data volume breaks down for high-cardinality keyed workloads, round-robin or custom partitioners, and mixed workloads, forcing buffer.memory to be sized as active_partition_count x batch.size — an increasingly costly tradeoff on object-storage-backed clusters whose higher produce latency demands a larger batch.size.

Details