KIP-95 — Incremental Batch Processing for Kafka Streams

Discarded Streams

Proposes an incremental batch processing mode for Kafka Streams where an application instance starts, processes all records up to the current end-of-log (EOL), then shuts down — analogous to a recurring Hadoop batch job. Kafka Streams in its default mode is designed for continuous stream processing, but many pipelines need periodic batch semantics (process everything accumulated since last run) that cannot be naturally expressed in a streaming topology.

Details