KIP-1266 — Bounding The Number Of RemoteLogMetadata Messages via Compacted RemoteLogMetadata Topic

Discussion Kafka 4.3 Tiered Storage

Converts the __remote_log_metadata internal topic to a compacted topic so that only the latest event per remote segment key is retained, bounding the topic's size and reducing broker startup time for rebuilding the in-memory RemoteLogMetadataCache. The topic currently grows unbounded as an append-only log of all remote segment lifecycle events, causing slow startup (full topic replay required) and high broker memory usage for clusters with large tiered storage histories.

Details