conduktor.io ↗

KIP-644 — Support sources which can't give out changed data in kafka connect

Discussion Connect

KIP-644 proposes extending the `SourceTask` contract to allow the task to return both output records and a state snapshot per `poll()` call, with Connect persisting the state to an internal topic and providing `SourceTaskContext` access to the previous state snapshot on restart. This enables sources that can only provide full snapshots (not change feeds) to be supported in Connect by having the framework compute the delta between the current and previous snapshots.

Details

AuthorNikhil Kumar
StatusDiscussion
WikiView on Apache Wiki
Created2020-07-16
Last Modified2020-07-16
Explore how this KIP affects the Kafka protocol in the Protocol Explorer, or see the full KIP database.