KIP-98 — Exactly Once Delivery and Transactional Messaging

Accepted Kafka 1.0 TransactionsProducer

Introduces exactly-once semantics (EOS) to Kafka via idempotent producers (deduplication via producer ID + sequence number) and multi-partition atomic transactions (committed via a two-phase protocol through the transaction coordinator). Kafka previously guaranteed at-least-once delivery, meaning producer retries on broker failures could produce duplicate records—a correctness problem for stream processing pipelines requiring exactly-once processing guarantees.

Protocol Impact

Produce · Fetch · OffsetCommit · FindCoordinator · AddPartitionsToTxn · AddOffsetsToTxn · EndTxn · WriteTxnMarkers · TxnOffsetCommit

Details