Kafka Exception MisconfiguredInternalTopicException
org.apache.kafka.streams.errors.MisconfiguredInternalTopicException
Non-retriable
Streams
MisconfiguredInternalTopicException is a Kafka client-side exception (org.apache.kafka.streams.errors.MisconfiguredInternalTopicException).
Common Causes
- A pre-existing internal topic (changelog or repartition, named <application.id>-<store/operator>-changelog / -repartition) has the WRONG partition count: Streams requires the changelog partition count to equal the task's input partition count, so 'expected: N; actual: M' fails validation in InternalTopicManager and the client shuts down.
- An internal topic has the wrong cleanup.policy: changelog topics must be compact (or compact,delete with retention) for state to be replayable; if someone manually created/altered the topic with cleanup.policy=delete, validation rejects it.
- The Kafka cluster (or its internal topics) was deleted and recreated with the same name but DEFAULT settings (e.g. broker num.partitions, cleanup.policy=delete), so the topics no longer match what the topology demands.
- Source topic was repartitioned (partition count changed) between runs while the changelog/repartition topics kept the old partitioning, so they no longer line up.
Solutions
- Stop ALL instances, then run kafka-streams-application-reset.sh to delete and let Streams recreate the internal topics with the correct partitions/config; or manually delete the stale <application.id>-*-changelog / -repartition topics so Streams recreates them.
- If you pre-create internal topics (managed/locked-down clusters), match exactly: partitions = input-topic partitions, cleanup.policy=compact for changelogs (compact,delete with retention.ms for windowed stores). Name operators explicitly (Materialized.as(...), Repartitioned.as(...)) so topic names are stable and pre-creatable.
- Never change the partition count of source topics under a running Streams app; if you must repartition, reset the application and reprocess.
- Verify the actual topic config before blaming the app: kafka-topics.sh --describe and kafka-configs.sh --describe to confirm partitions and cleanup.policy on each internal topic match the topology's expectation.
Example Stack Trace
org.apache.kafka.streams.errors.MisconfiguredInternalTopicException: Existing internal topic my-app-Counts-changelog has invalid partitions: expected: 1; actual: 3. Use 'kafka-streams-application-reset.sh' tool to clean up invalid topics before processing.
at org.apache.kafka.streams.processor.internals.InternalTopicManager.getNumPartitions(InternalTopicManager.java:380)
at org.apache.kafka.streams.processor.internals.InternalTopicManager.validateTopics(InternalTopicManager.java:300)
at org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:200)
at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.prepareTopic(StreamsPartitionAssignor.java:1180)
at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:560)Diagnostic Commands
kafka-topics.sh --bootstrap-server <b>:9092 --describe --topic <application.id>-<store>-changelog # check actual partition count vs input topic
kafka-configs.sh --bootstrap-server <b>:9092 --describe --entity-type topics --entity-name <application.id>-<store>-changelog # confirm cleanup.policy=compact
kafka-streams-application-reset.sh --application-id <app-id> --bootstrap-server <b>:9092 --input-topics <topics> # then restart to recreate internal topics correctlyRelated
Related Streams exceptions: BrokerNotFoundException · InternalTopicsAlreadySetupException · InvalidStateStoreException · InvalidStateStorePartitionException · LockException · MissingInternalTopicsException · MissingSourceTopicException · ProcessorStateException
Hitting
MisconfiguredInternalTopicException in production? Conduktor Console gives you real-time visibility into clients, consumer groups, and broker health. Browse every Kafka exception or protocol error code.