Kafka Exception UnknownProducerIdException
org.apache.kafka.common.errors.UnknownProducerIdException
Non-retriable
Producer
This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.
Common Causes
- The partition log start offset advanced past the last batch from this producer (for example because of retention, compaction, or DeleteRecords), so the broker no longer has producer state for the producer ID
- A leader change or log truncation removed the producer state snapshot on the new leader, so the broker cannot validate the next sequence number from this producer
- A long-idle idempotent or transactional producer resumed sending after its prior producer state had been aged out of the partition's retained history
Solutions
- Avoid aggressive log truncation on topics used by idempotent or transactional producers; review retention and any DeleteRecords workflow if producers resume after long idle periods
- On this error, create a new producer session. For transactional producers, recreate the producer and call initTransactions(); for idempotent non-transactional producers, close and recreate the producer before retrying
- If the producer can be idle for long periods, make sure the topic retains enough history that the leader still has producer state when the producer resumes
Example Stack Trace
org.apache.kafka.common.errors.UnknownProducerIdException: This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.Diagnostic Commands
# Look for related errors in broker logs
grep 'UNKNOWN_PRODUCER_ID\|UnknownProducerIdException\|ProducerStateManager' /path/to/kafka/logs/server.log | tail -30
# Check topic retention and cleanup settings
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name <topic> --describe | grep -E 'retention|cleanup.policy|min.compaction.lag|max.compaction.lag'Related
Protocol error: UNKNOWN_PRODUCER_ID (code 59)
Related Producer exceptions: BufferExhaustedException · InvalidProducerEpochException · OutOfOrderSequenceException · RecordBatchTooLargeException · RecordTooLargeException · TransactionAbortedException · UnsupportedForMessageFormatException
Hitting
UnknownProducerIdException in production? Conduktor Console gives you real-time visibility into clients, consumer groups, and broker health. Browse every Kafka exception or protocol error code.