Kafka Error KAFKA_STORAGE_ERROR
Error code 56 · Retriable Broker
Disk error when trying to access log file on the disk.
Common Causes
- Underlying disk is full, failing, or has I/O errors — OS returns errors when Kafka tries to write/read log segments
- Log directory mounted on NFS/remote storage with intermittent connectivity causing sporadic I/O failures
- File descriptor limits exhausted (`ulimit -n`) preventing Kafka from opening new segment files
Solutions
- Check disk health (`dmesg | grep -i error`, `smartctl -a /dev/sdX`) and free space (`df -h`); replace failing disk and restore from replica
- Increase `ulimit -n` to at least 100000 in the Kafka startup script or systemd unit (`LimitNOFILE=100000`)
- If a specific log directory is affected, use `kafka-reassign-partitions.sh` to move replicas away from the bad broker, then repair and recommission
Diagnostic Commands
# Look for disk I/O errors in kernel logs
dmesg | grep -iE 'error|fault|I/O' | tail -30
# Check for log directory errors on the broker
kafka-log-dirs.sh --bootstrap-server localhost:9092 --broker-list 0 --describe | grep -i errorRelated APIs
This error can be returned by: DescribeLogDirs · EndTxn · Fetch · InitProducerId · WriteTxnMarkers
Debugging Kafka errors? Conduktor Console gives you real-time visibility into your cluster. Explore all errors in the Error Decoder.