KIP-1044 — A proposal to change idempotent producer -- server implementation

Discussion ProducerBroker

Proposes changes to the idempotent producer's server-side PID (producer ID) tracking to address OOM failures caused by an unbounded explosion in the number of tracked PID/epoch pairs and to reduce the window between when a PID is received and when it is durably persisted. The fixed-size circular buffer design in the original KIP-98 spec was not implemented as specified, leading to unbounded memory growth in clusters with many concurrent transactional producers.

Details