Compacted Topics Need Explicit delete.retention.ms

Compacted topics must define delete.retention.ms in [1h, 30d].

Rationale

On compacted topics, tombstones (null-value records) survive for `delete.retention.ms` before they are physically removed. The default 86400000ms (24h) breaks consumers that fall behind for more than a day: they miss the delete and end up with ghost keys. Force teams to set this consciously.

Pattern

cleanup.policy contains "compact" → delete.retention.ms in [3600000, 2592000000]

Examples

Parameters

Governs

This policy relates to the following Kafka configuration keys:

• cleanup.policy
• delete.retention.ms
• min.compaction.lag.ms

Implementation

Drop this YAML into Conduktor Console as a ResourcePolicy, then link it from an ApplicationInstance, Application, or KafkaCluster.

# Conduktor self-service ResourcePolicy
---
apiVersion: self-serve/v1
kind: ResourcePolicy
metadata:
  name: compact-topic-requires-tombstone-retention
spec:
  targetKind: Topic
  description: compacted topics must declare delete.retention.ms in [1h, 30d]
  rules:
    - condition: '!("cleanup.policy" in spec.configs) || !spec.configs["cleanup.policy"].contains("compact") || ("delete.retention.ms" in spec.configs && int(string(spec.configs["delete.retention.ms"])) >= 3600000 && int(string(spec.configs["delete.retention.ms"])) <= 2592000000)'
      errorMessage: "compacted topics must declare delete.retention.ms in [1h, 30d] — default 24h often breaks slow consumers"