KIP-1159 — Large message reference based Serializer

Discussion ClientProducer

Proposes a reference-based Serializer/Deserializer that transparently stores large payloads in external object storage (e.g., S3) and embeds only the storage reference in the Kafka message. Kafka's hard limit on message.max.bytes cannot be raised indefinitely without causing broker memory, network, and stability issues, and existing workarounds (chunking, manual reference patterns) require complex application-level logic.

Details