KIP-526 — Reduce Producer Metadata Lookups for Large Number of Topics

Discussion Kafka 2.5 Producer

Proposes allowing producers to pre-declare a set of known topics via a new `pre.known.topics` configuration, enabling the producer to batch metadata requests for all those topics at startup rather than fetching metadata on-demand per topic as records are first produced. For producers writing to a large, stable set of topics (e.g., thousands of per-tenant topics), on-demand metadata fetching causes high latency for first records to each new topic as individual metadata round-trips serialize.

Details