Kafka Exception MissingInternalTopicsException
org.apache.kafka.streams.errors.MissingInternalTopicsException
Non-retriable
Streams
MissingInternalTopicsException is a Kafka client-side exception (org.apache.kafka.streams.errors.MissingInternalTopicsException).
Common Causes
- The Streams app's principal lacks ACLs to create internal topics: the embedded AdminClient needs CREATE on the cluster (or topic prefix) plus READ/WRITE on the changelog/repartition topics. Without them, creation of <application.id>-*-changelog / -repartition fails and the topics are reported missing.
- A managed/locked-down Kafka that forbids dynamic topic creation: Azure Event Hubs, Aiven, and MSK Serverless/Express reject the AdminClient create call (Event Hubs blocks it outright; MSK Express/Serverless protect segment.bytes which Streams hard-codes to ~50MB), so the internal topics never come into existence.
- Internal (changelog/repartition) topics were deleted out from under a running or restarting application, so on the next rebalance the assignor finds them missing.
- Cluster was wiped/recreated, or the app was moved to a fresh cluster, while local state/offsets still reference internal topics that don't exist on the new cluster.
Solutions
- Grant the Streams principal the right ACLs: kafka-acls.sh --add --allow-principal User:<app> --operation Create --cluster; plus --operation Read --operation Write on the <application.id>-* internal topics (and the consumer group).
- On platforms that block dynamic topic creation (Event Hubs / Aiven / MSK Serverless-Express), PRE-CREATE every internal topic yourself with the correct partitions, naming operators explicitly so the names are deterministic (Materialized.as / Repartitioned.as). Use cleanup.policy=compact ONLY for changelog topics; repartition topics must stay cleanup.policy=delete — compacting a repartition topic silently drops records.
- Do not delete internal topics of a running app; if internal topics were lost, stop the app and run kafka-streams-application-reset.sh, then restart so they are recreated.
- If on a managed broker, follow its docs for the protected configs (e.g. MSK Express: create internal topics manually because segment.bytes is protected) rather than relying on Streams' auto-create.
Example Stack Trace
org.apache.kafka.streams.errors.MissingInternalTopicsException: Internal topics are missing: [my-app-Counts-changelog, my-app-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition]
at org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:215)
at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.prepareTopic(StreamsPartitionAssignor.java:1180)
at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:560)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onLeaderElected(ConsumerCoordinator.java:707)
Caused by: org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [my-app-Counts-changelog]Diagnostic Commands
kafka-acls.sh --bootstrap-server <b>:9092 --list --principal User:<app-principal> # verify Create + Read/Write on internal topics
kafka-topics.sh --bootstrap-server <b>:9092 --list | grep <application.id> # see which internal topics actually exist
kafka-topics.sh --bootstrap-server <b>:9092 --create --topic <application.id>-<store>-changelog --partitions <N> --config cleanup.policy=compact # pre-create on locked-down clustersRelated
Related Streams exceptions: BrokerNotFoundException · InternalTopicsAlreadySetupException · InvalidStateStoreException · InvalidStateStorePartitionException · LockException · MisconfiguredInternalTopicException · MissingSourceTopicException · ProcessorStateException
Hitting
MissingInternalTopicsException in production? Conduktor Console gives you real-time visibility into clients, consumer groups, and broker health. Browse every Kafka exception or protocol error code.