conduktor.io ↗

KIP-793 — Allow sink connectors to be used with topic-mutating SMTs

Accepted Kafka 3.6 Connect

Adds originalTopic, originalKafkaPartition, and originalKafkaOffset fields to SinkRecord so that sink tasks overriding preCommit() can return the pre-transform offsets that the framework expects. When a topic-mutating SMT (e.g., RegexRouter) changes the topic name, the connector only sees the post-transform topic/partition/offset, causing preCommit() to return wrong offsets and breaking at-least-once offset tracking.

Details

AuthorDiego Erdody
StatusAccepted
Kafka Version3.6
JIRAKAFKA-13431
WikiView on Apache Wiki
Created2021-11-03
Last Modified2023-07-07
Explore how this KIP affects the Kafka protocol in the Protocol Explorer, or see the full KIP database.