conduktor.io ↗

KIP-213 — Support non-key joining in KTable

Accepted Kafka 2.4 Streams

Adds foreign-key KTable-to-KTable joins to the Kafka Streams DSL through `join`/`leftJoin` overloads taking a `foreignKeyExtractor`, implemented with subscription records over repartition topics, a `CombinedKey` (primary key plus foreign key) prefix-scan store on the right-hand table, and an original-value hash carried round-trip so stale results are discarded when the extracted foreign key changes. KTables could previously only be joined on their primary key, so the one-to-many relationships that arrive from CDC pipelines (JDBC connect, Debezium, Maxwell) had to be resolved with a poorly supported group-by/join/lateral-view workaround at odds with record-based processing.

Details

AuthorJan Filipiak
StatusAccepted
Kafka Version2.4
JIRAKAFKA-3705
WikiView on Apache Wiki
Created2017-10-25
Last Modified2021-05-21
Explore how this KIP affects the Kafka protocol in the Protocol Explorer, or see the full KIP database.