KIP-1327 — Prevent Hot Data Loss on Partition Expansion for Reset Offset Policy

Discussion Consumer

Proposes an auto.offset.reset.new.partitions consumer config that applies a separate reset policy to partitions created after the consumer group, decoupling them from the policy used for pre-existing partitions. With auto.offset.reset=latest, records written to newly added partitions during the metadata blindness window between partition creation and the consumer's next metadata refresh are skipped with no exception, warning, or log entry, turning routine partition expansion into silent data loss.

Protocol Impact

ListOffsets · Metadata · Heartbeat · CreatePartitions · DeleteGroups · ConsumerGroupHeartbeat · ShareGroupHeartbeat · InitializeShareGroupState · ReadShareGroupState · StreamsGroupHeartbeat

Details