KIP-1121 — Compression acceleration in Kafka - Apache Kafka - Apache Software Foundation

Discussion BrokerProducer

Proposes a hardware acceleration framework for Kafka's existing compression algorithms (gzip, zstd, snappy, lz4), allowing producers, brokers, and consumers to offload compression/decompression to hardware accelerators (e.g., Intel QAT, GPU) without changing the wire format. Compression improves throughput and reduces storage costs but adds CPU latency; hardware accelerators can reduce that latency without sacrificing compression ratio, but Kafka has no pluggable acceleration abstraction.

Details