Kafka Exception MissingSourceTopicException
org.apache.kafka.streams.errors.MissingSourceTopicException
Non-retriable
Streams
MissingSourceTopicException is a Kafka client-side exception (org.apache.kafka.streams.errors.MissingSourceTopicException).
Common Causes
- A source topic (or an intermediate through()/repartition topic that an older Streams version auto-created) does not exist at rebalance time: the leader encodes INCOMPLETE_SOURCE_TOPIC_METADATA into the assignment and each thread throws 'One or more source topics were missing during rebalance'.
- Upgrade regression: pre-2.x Streams tolerated intermediate topics being created lazily as processing progressed; newer versions require source topics to be pre-created and fail fast instead.
- A typo / wrong name in stream(...) / table(...), or the topic lives on a different cluster than the configured bootstrap.servers, so it's genuinely unknown to the assignor.
- A source topic was deleted while the application was running (KIP-662 makes this a hard error rather than a silent hang).
Solutions
- Pre-create every source topic (and any external intermediate topic you reference) BEFORE starting the app, exactly as the assignor's error advises: 'please make sure all source topics have been pre-created'. Don't rely on broker auto-create for source topics.
- After an upgrade from 1.x, audit the topology for topics that used to be created lazily (intermediate output topics fed back as sources) and create them explicitly, or use the Streams DSL so Streams manages repartition topics itself.
- Verify the exact topic names with kafka-topics.sh --list and confirm bootstrap.servers points at the cluster that actually hosts them; fix typos in stream()/table()/Consumed names.
- Set an uncaught-exception handler (StreamsUncaughtExceptionHandler) so a MissingSourceTopicException either shuts the client down cleanly or replaces the thread, and surface a clear operator alert. Check broker logs for 'Source topic ... is missing/unknown during rebalance'.
Example Stack Trace
org.apache.kafka.streams.errors.MissingSourceTopicException: One or more source topics were missing during rebalance
at org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onAssignment(StreamsRebalanceListener.java:107)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.invokeOnAssignment(ConsumerCoordinator.java:284)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:449)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:740)
(leader-side log: 'Source topic output-topic-2 is missing/unknown during rebalance ... Returning error INCOMPLETE_SOURCE_TOPIC_METADATA')Diagnostic Commands
kafka-topics.sh --bootstrap-server <b>:9092 --list | grep <source-topic> # confirm the source topic actually exists on this cluster
kafka-topics.sh --bootstrap-server <b>:9092 --create --topic <source-topic> --partitions <N> --replication-factor <R> # pre-create before starting Streams
grep -i 'INCOMPLETE_SOURCE_TOPIC_METADATA\|missing/unknown during rebalance' streams.log # find which topic the leader flaggedRelated
Related Streams exceptions: BrokerNotFoundException · InternalTopicsAlreadySetupException · InvalidStateStoreException · InvalidStateStorePartitionException · LockException · MisconfiguredInternalTopicException · MissingInternalTopicsException · ProcessorStateException
Hitting
MissingSourceTopicException in production? Conduktor Console gives you real-time visibility into clients, consumer groups, and broker health. Browse every Kafka exception or protocol error code.