What causes CorruptRecordException?

Log segment or index corruption on disk after storage failure, filesystem corruption, or an unclean shutdown damaged local files Producer sent a record with a null key to a compacted topic (cleanup.policy includes compact), which Kafka rejects as an invalid/corrupt record A buggy or non-standard client produced a malformed record batch (bad CRC, bad length fields, or invalid compressed batch wrapper)

How do I fix CorruptRecordException?

If broker logs mention a null key on a compacted topic, fix the producer to always set a key for that topic or remove compaction from the topic's cleanup.policy If broker logs mention CRC or log corruption, identify the bad replica with kafka-dump-log.sh or broker logs, stop the affected broker, remove only the corrupt replica copy after confirming another healthy replica exists, and let replication rebuild it If only one client or library triggers this error, compare its compression.type and client version with a standard Kafka client and upgrade or replace that client before retrying

Is CorruptRecordException retriable?

Yes — CorruptRecordException is a subclass of RetriableException, so Kafka clients can safely retry; it usually clears after a short delay.

Kafka Exception `CorruptRecordException`

org.apache.kafka.common.errors.CorruptRecordException

Retriable Broker

This exception indicates a record has failed its internal CRC check, this generally indicates network or disk corruption.

Common Causes

Log segment or index corruption on disk after storage failure, filesystem corruption, or an unclean shutdown damaged local files
Producer sent a record with a null key to a compacted topic (cleanup.policy includes compact), which Kafka rejects as an invalid/corrupt record
A buggy or non-standard client produced a malformed record batch (bad CRC, bad length fields, or invalid compressed batch wrapper)

Solutions

If broker logs mention a null key on a compacted topic, fix the producer to always set a key for that topic or remove compaction from the topic's cleanup.policy
If broker logs mention CRC or log corruption, identify the bad replica with kafka-dump-log.sh or broker logs, stop the affected broker, remove only the corrupt replica copy after confirming another healthy replica exists, and let replication rebuild it
If only one client or library triggers this error, compare its compression.type and client version with a standard Kafka client and upgrade or replace that client before retrying

Example Stack Trace

org.apache.kafka.common.errors.CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, has a null key for a compacted topic, or is otherwise corrupt.

Diagnostic Commands

# Scan log segment for corruption or errors
kafka-dump-log.sh --files /var/kafka-logs/<topic>-<partition>/00000000000000000000.log --print-data-log 2>&1 | grep -i 'corrupt\|invalid\|error'

# Inspect log directories on the broker
kafka-log-dirs.sh --bootstrap-server localhost:9092 --topic-list <topic> --describe

Protocol error: CORRUPT_MESSAGE (code 2)

Hitting CorruptRecordException in production? Conduktor Console gives you real-time visibility into clients, consumer groups, and broker health. Browse every Kafka exception or protocol error code.

Kafka Exception CorruptRecordException

Common Causes

Solutions

Example Stack Trace

Diagnostic Commands

Related

Kafka Exception `CorruptRecordException`