KIP-631 — The Quorum-based Kafka Controller

Discussion Kafka 2.8 KRaft

Details the architectural changes to the Kafka Controller when operating in KRaft (KIP-500) mode: the controller becomes a Raft-based quorum member storing metadata in an internal `__cluster_metadata` topic, eliminating ZooKeeper as the metadata store. The existing ZooKeeper-dependent controller architecture is a scalability and operational bottleneck; the KRaft controller enables faster failover, larger partition counts, and removal of the ZooKeeper dependency.

Protocol Impact

Fetch · Metadata · LeaderAndIsr · StopReplica · UpdateMetadata · BrokerRegistration · UnregisterBroker

Details