Kafka Error `UNKNOWN_PRODUCER_ID`

Error code 59 · Non-retriable Producer

This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.

Common Causes

The partition log start offset advanced past the last batch from this producer (for example because of retention, compaction, or DeleteRecords), so the broker no longer has producer state for the producer ID
A leader change or log truncation removed the producer state snapshot on the new leader, so the broker cannot validate the next sequence number from this producer
A long-idle idempotent or transactional producer resumed sending after its prior producer state had been aged out of the partition's retained history

Solutions

Avoid aggressive log truncation on topics used by idempotent or transactional producers; review retention and any DeleteRecords workflow if producers resume after long idle periods
On this error, create a new producer session. For transactional producers, recreate the producer and call initTransactions(); for idempotent non-transactional producers, close and recreate the producer before retrying
If the producer can be idle for long periods, make sure the topic retains enough history that the leader still has producer state when the producer resumes

Diagnostic Commands

# Look for related errors in broker logs
grep 'UNKNOWN_PRODUCER_ID\|UnknownProducerIdException\|ProducerStateManager' /path/to/kafka/logs/server.log | tail -30

# Check topic retention and cleanup settings
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name <topic> --describe | grep -E 'retention|cleanup.policy|min.compaction.lag|max.compaction.lag'

Related APIs

This error can be returned by: Produce

Related KIPs

KIP-360

Debugging Kafka errors? Conduktor Console gives you real-time visibility into your cluster. Explore all errors in the Error Decoder.