log.cleaner.dedupe.buffer.size — Kafka Broker Configuration
The total memory used for log deduplication across all cleaner threads.
Description
The total memory used for log deduplication across all cleaner threads
Default Values by Kafka Version
| Kafka Version | Default Value |
|---|---|
| 0.8.1 | 500*1024*1024 |
| 0.8.2 | 500*1024*1024 |
| 0.9.0 | 134217728 |
| 0.10.0 | 134217728 |
| 0.10.1 | 134217728 |
| 0.10.2 | 134217728 |
| 0.11.0 | 134217728 |
| 1.0 | 134217728 |
| 1.1 | 134217728 |
| 2.0 | 134217728 |
| 2.1 | 134217728 |
| 2.2 | 134217728 |
| 2.3 | 134217728 |
| 2.4 | 134217728 |
| 2.5 | 134217728 |
| 2.6 | 134217728 |
| 2.7 | 134217728 |
| 2.8 | 134217728 |
| 3.0 | 134217728 |
| 3.1 | 134217728 |
| 3.2 | 134217728 |
| 3.3 | 134217728 |
| 3.4 | 134217728 |
| 3.5 | 134217728 |
| 3.6 | 134217728 |
| 3.7 | 134217728 |
| 3.8 | 134217728 |
| 3.9 | 134217728 |
| 4.0 | 134217728 |
| 4.1 | 134217728 |
| 4.2 | 134217728 |
Tuning Recommendation
| Profile | Recommended | Why |
|---|---|---|
| broker / cost | 268435456 | Doubles the log compaction deduplication buffer from 128MB to 256MB. This memory is split evenly across all cleaner threads (e.g. 2 threads = 128MB each). Larger per-thread buffer means fewer compaction passes for topics with high key cardinality, reducing I/O. |
Related Configs
log.cleaner.threads · log.cleaner.min.cleanable.ratio · log.cleanup.policy
Manage Kafka configs across all your clusters with Conduktor Console — view, compare, and update configurations in one place.