KIP-852 — Optimize calculation of size for log in remote tier
Accepted Kafka 3.6 Tiered StorageMetrics
Adds a RemoteLogMetadataManager#remoteLogSize(TopicIdPartition) API and a RemoteLogSizeBytes metric, allowing RLMM implementations to compute total remote tier size without listing all segment metadata. The existing O(num_remote_segments) scan via listRemoteLogSegments() becomes prohibitively expensive as the number of remote segments grows, slowing the retention enforcement loop.
Details
| Author | Divij Vaidya |
| Status | Accepted |
| Kafka Version | 3.6 |
| JIRA | KAFKA-14038 |
| Wiki | View on Apache Wiki |
| Created | 2022-07-01 |
| Last Modified | 2023-07-26 |
Explore how this KIP affects the Kafka protocol in the Protocol Explorer, or see the full KIP database.