conduktor.io ↗

KIP-1075 — Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Accepted Kafka 4.0 Tiered StorageBroker

KIP-1075 introduces a delayed purgatory (`DelayedRemoteListOffsets`) and a new `TimeoutMs` field in `ListOffsetsRequest` so timestamp-based offset lookups that require fetching remote indexes are handled asynchronously by dedicated remote-log-reader threads rather than blocking request-handler threads. `ListOffsets` requests against remote-storage-enabled topics require non-deterministic latency to fetch remote offset and time indexes; when many such requests arrive concurrently, request-handler thread exhaustion starves higher-priority `FETCH` and `PRODUCE` requests.

Protocol Impact

ListOffsets v10

Details

AuthorKamal Chandraprakash
StatusAccepted
Kafka Version4.0
JIRAKAFKA-15859
WikiView on Apache Wiki
Created2024-08-02
Last Modified2024-08-29
Explore how this KIP affects the Kafka protocol in the Protocol Explorer, or see the full KIP database.