KIP-227 — Introduce Incremental FetchRequests to Increase Partition Scalability
Accepted Kafka 1.1 ProtocolConsumer
Introduces incremental FetchRequests where followers only send partition-level deltas (additions/removals) rather than the full partition list on every fetch, and allows leaders to omit unchanged partition metadata from responses. Sending the full partition set in every FetchRequest and FetchResponse creates O(partitions) overhead that scales poorly as partition count grows into the hundreds of thousands.
Protocol Impact
Details
| Author | Colin McCabe |
| Status | Accepted |
| Kafka Version | 1.1 |
| JIRA | KAFKA-6254 |
| Wiki | View on Apache Wiki |
| Created | 2017-11-21 |
| Last Modified | 2022-09-11 |
Explore how this KIP affects the Kafka protocol in the Protocol Explorer, or see the full KIP database.