Kafka Improvement Proposals (KIPs) Database

TLDR: Adds a role tag (broker, controller, or broker+controller) to the metric labels emitted by NodeToControllerChannelManagerImpl so that SelectorMetrics can be disambiguated by node role in KRaft mode. The component currently tags metrics only with broker-id, producing duplicate or misleading metrics on controller-only or combined nodes where brokerId is not meaningful.

TLDR: Introduces metadata-version-aware configuration validation, so that broker config constraints can be tied to the cluster's active MetadataVersion and enforced dynamically as the cluster upgrades. Current ConfigDef validation is version-agnostic and runs statically at startup, preventing safe introduction of configs that are only valid above a certain MetadataVersion.

TLDR: Introduces a dedicated MirrorMaker 2 connector for consumer group offset synchronization that is separate from the existing MirrorCheckpointConnector, providing real-time offset sync via the consumer offsets topic rather than periodic checkpoint emission. MirrorCheckpointConnector conflates checkpoint generation with offset sync and commits offsets only periodically, creating a lag window during which a failover to the target cluster results in reprocessing.

TLDR: Introduces rack-aware min.insync.replicas enforcement at produce time, rejecting writes unless ISR acknowledgments come from replicas spanning a configurable minimum number of distinct racks. The existing min.insync.replicas counts total ISR replicas without regard to rack placement, meaning all acknowledging replicas could reside in the same availability zone, providing no durability guarantee against a single-AZ failure.

TLDR: Adds transactional acknowledgment support for share group consumers, ensuring that a record's side-effect writes and its share group acknowledgment are committed atomically within a single Kafka transaction. Without this, a framework processing share group records can produce output and then fail before acknowledging, causing re-delivery and double-processing with no EOS guarantee.

TLDR: Adds opt-in SSL hot reload to Kafka clients by monitoring keystore and truststore files for changes and automatically reconfiguring the SSL context without restarting the client. Currently, SSL credentials are loaded once at startup and never refreshed, so certificate rotation or expiry requires restarting every client—a significant operational burden in environments using short-lived certificates.

TLDR: Enforces partition assignment validation in the broker for KafkaProducer.sendOffsetsToTransaction(), rejecting transactional offset commits for partitions not assigned to the requesting consumer group member. The broker currently verifies the member's epoch but does not check whether the committed partitions are actually assigned to that member, allowing a valid-epoch member to maliciously or accidentally overwrite offsets for partitions it does not own.

TLDR: Wires KIP-1271's headers-aware state store implementations into the Kafka Streams DSL via an explicit opt-in mechanism, allowing operators like table(), aggregate(), and join() to use header-preserving stores. Without this DSL integration, developers using the high-level API had no path to adopt header-aware stores even after KIP-1271 provided the underlying interfaces.

TLDR: Introduces CloseOptions.DEFAULT as an explicit enum value for KafkaStreams.close() that selects the correct shutdown behavior based on the active rebalance protocol (LEAVE_GROUP for the Streams Protocol, REMAIN_IN_GROUP for Classic). Without this, KafkaStreams.close() with no CloseOptions implicitly uses REMAIN_IN_GROUP, which is semantically wrong for the KIP-1071 Streams Protocol where dynamic membership makes remaining in the group after shutdown counterproductive.

TLDR: Updates the public contract of KafkaStreams#cleanUp to explicitly preserve the process identity metadata file (kafka-streams-process-metadata) and the application directory lock file (.lock), clarifying that the method clears local state but may retain the application directory. The existing behavior incorrectly attempted to delete the application directory even when valid metadata files were present, generating misleading warning logs.

TLDR: Introduces a new auto.offset.reset policy called partition-start that resets consumers to the beginning of only newly discovered partitions while continuing from committed offsets on existing ones. The existing earliest policy forces full historical replay and latest silently drops messages produced between partition discovery and consumer assignment—neither is safe during partition expansion or initial consumer startup.

TLDR: Updates MirrorSourceConnector and MirrorCheckpointConnector to register their metrics via the KIP-877 connector metrics API instead of creating private Metrics instances with their own MetricsReporter instances. The previous workaround—each connector instantiating its own Metrics object—bypassed standard metrics plumbing and was never available for MirrorHeartbeatConnector at all.

TLDR: Cluster Mirroring introduces native broker-side cross-cluster replication as a built-in Kafka feature, replacing the need for external MirrorMaker 2 (MM2) Connect workers. MM2's standalone deployment model creates significant operational burden—separate provisioning, independent lifecycle management, and redundant compression/decompression overhead—which this KIP eliminates by integrating replication directly into brokers.

TLDR: Adds configurable retry parameters (max retries, backoff, timeout) to KafkaStatusBackingStore in Kafka Connect for handling failures when writing connector and task status to the internal status topic. Currently, some write paths block indefinitely on broker unavailability while others silently drop status updates, making connector state persistence unreliable and unobservable during broker outages.

TLDR: Introduces native delayed message delivery to Kafka, allowing producers to set a future delivery timestamp on records so they are held by the broker and only made visible to consumers at the specified time. Kafka has no built-in scheduling mechanism; teams implementing delayed retry logic for share groups (or any queue workload) must build external scheduling infrastructure or poll-and-republish loops.

TLDR: Deprecates the Classic (pre-KIP-848) consumer rebalance protocol in KafkaConsumer with removal planned for a future major version, making the new incremental rebalance protocol (KIP-848) the only supported protocol. The Classic protocol's eager stop-the-world rebalances and thick client-side assignor logic are known operational pain points; the new protocol has been production-ready since Kafka 4.0 and the Classic implementation carries ongoing maintenance cost with diminishing benefit.

TLDR: Introduces a shared Configurable interface with a config() method across all Kafka Connect pluggable component types (Connector, Converter, Transformation, Predicate), and adds a REST endpoint to expose component configuration metadata. Each component type currently defines config() independently with no common supertype, making it impossible to generically discover and document configuration for all plugin types.

TLDR: Extends tiered storage (KIP-405) to support compacted topics by handling the semantics of compaction in remote object storage, including uploading compacted segments, tracking which keys have been compacted away, and serving offset lookups over compacted remote segments. KIP-405 only supports append-only (delete-retention) log segments; compacted topics are a fundamentally different retention model that requires tracking the latest value per key across both local and remote tiers.

TLDR: Extends Kafka Streams state stores to persist record headers alongside key and value bytes, introducing interfaces like TimestampedKeyValueStoreWithHeaders and VersionedKeyValueStoreWithHeaders. Currently, headers are stripped when records enter stateful operators, breaking header-dependent serdes (e.g., Schema Registry header-based schema-id format), tracing propagation, and any other header-carried semantics.

TLDR: Extends the ProcessingExceptionHandler to cover GlobalStreamThread processors (GlobalKTable), with a new processing.exception.handler.global.enabled boolean config to opt in. Currently any exception in GlobalKTable record processing kills the GlobalStreamThread and shuts down the entire Streams application, unlike regular KStream/KTable processing where the handler can recover gracefully.

TLDR: Introduces a configurable broker-side limit on the number of producer batch sequence numbers retained for idempotent deduplication, replacing the hardcoded constant ProducerStateEntry#NUM_BATCHES_TO_RETAIN (currently 5). This hardcoded value forces max.in.flight.requests.per.connection to be ≤5 for idempotent producers, artificially limiting producer throughput.

TLDR: Prevents partition count expansion for internal Kafka topics (__transaction_state, __consumer_offsets, __share_group_state) by blocking AdminClient alter-partition-count operations on them. These topics use hash-based routing of transaction IDs or group IDs to specific partitions; changing partition count breaks the routing invariant and causes coordinators to lose track of existing state, leading to data loss or correctness failures.

TLDR: Adds per-topic and per-consumer-group tiered storage cost attribution metrics (remote bytes fetched, remote API call counts) to enable operators to charge back object storage costs to individual tenants. With tiered storage, remote fetch operations (S3 GetObject calls) incur variable per-request costs, but Kafka currently exposes no metrics to attribute those costs to specific topics or consumer groups.

TLDR: Converts the __remote_log_metadata internal topic to a compacted topic so that only the latest event per remote segment key is retained, bounding the topic's size and reducing broker startup time for rebuilding the in-memory RemoteLogMetadataCache. The topic currently grows unbounded as an append-only log of all remote segment lifecycle events, causing slow startup (full topic replay required) and high broker memory usage for clusters with large tiered storage histories.

TLDR: Fixes the RemoteIndexCache TTL eviction logic so that stale and expired tiered storage index entries (offset, time, transaction, producer snapshot indexes) are actually evicted from the cache. A bug in the current implementation prevents TTL-based eviction from triggering correctly, causing the cache to grow unbounded with stale entries and eventually consuming excessive heap or causing repeated unnecessary remote fetches.

TLDR: Moves consumer group partition assignment calculation from a synchronous, per-heartbeat computation to a batched, asynchronous offload executed by a dedicated thread pool in the group coordinator. The KIP-848 server-side assignor runs assignment on the broker's main thread during heartbeat processing, which for large groups (thousands of members) can block for hundreds of milliseconds and degrade broker latency.

TLDR: Enables KRaft brokers to auto-format their log directories on first startup, eliminating the mandatory pre-start kafka-storage format step and automatically generating or joining a cluster ID. The current requirement forces operators to run a separate formatting command and supply a --cluster-id before any node can start, complicating automated and containerized deployments.

TLDR: Adds a startup configuration flag (state.dir.wipe.on.startup or equivalent) that causes Kafka Streams to delete all local state store data and checkpoint files before restoring from changelogs. A zombie-data edge case occurs when an instance restarts with stale local state after the changelog's delete.retention.ms has expired: the local checkpoint offset still exists on the broker but the corresponding changelog records have been deleted, causing silent state corruption on restore.

TLDR: Adds support for OAuth 2.0 client assertion (JWT Bearer, per RFC 7521/7523) as an authentication method for the client_credentials grant in Kafka's OAUTHBEARER SASL implementation. The current implementation only supports client_secret via HTTP Basic authentication (KIP-768), which requires sharing a long-lived secret—a security liability in zero-trust, short-lived-credential environments.

TLDR: Adds partition-level storage utilization percentage metrics that express current log size as a fraction of configured retention limits (both size-based and time-based local retention, and remote retention for tiered storage). Operators must currently compute these percentages manually by correlating size metrics with retention configs, and the calculation is especially complex for tiered topics with separate local/remote retention constraints.

TLDR: Aligns the behavior of Admin.incrementalAlterConfigs between broker-routed and controller-routed requests for null value handling, validation, and idempotency. When connecting via bootstrap.servers, brokers reject null values on non-DELETE operations; when connecting via bootstrap.controllers, controllers accept them silently—creating inconsistent semantics depending on the client's connection target.

TLDR: Proposes a Remote Read Replica (RRR) broker role that serves historical reads exclusively from tiered remote storage, decoupling cold-read traffic from hot-path leader/ISR brokers and enabling per-AZ RRR deployments to reduce cross-AZ latency and cost. Leader replicas currently handle all reads regardless of data age, causing I/O contention, elevated local storage requirements, and cross-AZ fetch costs for consumers reading historical offsets from tiered storage.

TLDR: Describes the client-side protocol changes required for Kafka consumers to fetch data directly from remote tiered storage, complementing the broker-side protocol introduced in KIP-1248. When brokers redirect consumers to tiered storage, consumers must resolve the remote segment location, authenticate to the object store, parse Kafka log segment format from raw object storage bytes, and handle transactions correctly—none of which the current consumer client supports.

TLDR: Proposes a TopologyValidator utility for Kafka Streams that detects breaking topology changes between deployments — such as internal topic renames, state store name shifts from added source nodes, or changed source/sink topics — before they cause data loss or silent state reset. Kafka Streams auto-generates incremental identifiers for unnamed nodes and stores, so inserting a new source node early in the topology silently shifts all downstream node names, reassigning changelog and repartition topics.

TLDR: Adds an assignment epoch to the new consumer group protocol (KIP-848) so that OffsetCommit and AddOffsetsToTransaction requests can be validated against the member's current assignment, fencing zombie commits from members whose partitions have been reassigned. The KIP-848 protocol uses member epochs for heartbeat fencing but did not carry epoch information in offset commit or transactional offset commit paths, leaving a window for stale members to overwrite offsets of their successors.

TLDR: Adds a num-keys metric to in-memory Kafka Streams state stores (InMemoryKeyValueStore, InMemorySessionStore, etc.) tracking the current number of entries. RocksDB stores expose an estimated key count metric, but in-memory stores emit no equivalent, leaving operators without visibility into memory pressure from state store growth.

TLDR: Enables consumer group offset resets while the group is actively consuming by introducing a new ResetOffset admin API that the group coordinator processes atomically with assignment reconciliation. Currently, offset resets require the consumer group to be fully stopped, forcing production services offline or requiring complex custom control planes to coordinate safe online resets.

TLDR: Promotes the Bytes class from org.apache.kafka.common.utils (internal package) to the public Kafka API surface. Bytes is already referenced in public Kafka Streams interfaces (e.g., KeyValueStore<Bytes, byte[]>), meaning users can depend on it through public APIs but cannot import it without using an internal package, creating a usability gap.

TLDR: Deprecates the backdoor mechanism that allows any client with clientId __admin_client to produce to internal topics (e.g. __consumer_offsets, __transaction_state) without authorization checks, with removal planned for Kafka 5.0. This bypass poses a security and operational risk: malicious or misconfigured clients can corrupt internal topic state with no differentiation from legitimate admin tooling.

TLDR: Moves validation of the application.server config (must be <host>:<port> format) from KafkaStreams construction time to StreamsConfig construction time, enabling faster fail detection. The check currently happens at client construction, which is too late for frameworks that create StreamsConfig independently and want to surface configuration errors before instantiating the streams instance.

TLDR: Introduces misrouted-connection detection by having the broker send a DisconnectResponse (or equivalent signal) when a client connects to the wrong broker, enabling the client to refresh metadata and reconnect correctly. In Kubernetes rolling restarts, a broker can change its advertised hostname and a client using stale bootstrap metadata silently connects to the wrong broker, leading to confusing failures that require a full client restart to resolve.

TLDR: Adds a configurable delayed upload option for tiered storage, deferring segment upload to remote storage until the segment ages beyond a threshold relative to the local retention period. The current implementation eagerly uploads all closed segments to remote storage immediately, storing the same data on both local disk and remote object storage simultaneously and incurring unnecessary storage costs when data is still within its local retention window.

TLDR: Introduces per-group dynamic configuration for share groups, including max delivery attempts and delivery timeout, superseding the global broker-level config for these parameters. KIP-932 fixed share group behavior at the broker level without group-level overrides, making it impossible to tune error tolerance independently for different share groups on the same cluster.

TLDR: Extends TopologyTestDriver to support multi-partition topics, enabling unit tests that exercise repartitioning, co-partitioned joins, and other partition-dependent stream processing logic. The driver currently simulates all topics as single-partition, making it impossible to test topologies that rely on repartition semantics without spinning up a full EmbeddedKafkaCluster.

TLDR: Enforces a minimum lower bound of 1000ms for controller.quorum.fetch.timeout.ms to preserve the timing invariant that fetch timeout must be at least 2× the Raft maximum fetch wait time (500ms). Operators setting an unusually small fetch timeout could violate the invariant, causing the KRaft follower to prematurely transition to Prospective state and trigger spurious elections even when the leader is healthy.

TLDR: Adds a new broker config remote.log.metadata.topic.min.isr (default 2) that is applied as min.insync.replicas when creating the __remote_log_metadata topic. Without an explicit min.isr, the topic defaulted to 1, meaning a single-broker failure in a 3-replica setup could cause tiered storage metadata loss and render remote data unreachable even though it still exists in object storage.

TLDR: Moves --bootstrap-server/--bootstrap-controller from a globally required argument group in kafka-feature.sh to per-subcommand required args, and adds a --unstable-feature-versions flag to the version-mapping subcommand. Offline subcommands like version-mapping were blocked from running without unnecessary cluster connection arguments, and unstable metadata versions were not queryable from the CLI.

TLDR: Proposes enforcing maximum length limits on Kafka resource names and identifiers (topic names, consumer group IDs, transactional IDs, client IDs, etc.). Without length limits, extremely long identifiers can inflate group coordinator data structures, cause unreadable CLI output, degrade broker performance, or be exploited for denial-of-service attacks.

TLDR: Adds a Kafka Streams config allow.os.group.write.access (default false) that grants Unix group write permission on the state directory when enabled. Streams hard-coded state directory permissions to owner-only write, blocking multi-user or group-based deployments where shared write access is a legitimate operational requirement.

TLDR: Adds a transactionVersion field to WriteTxnMarkersRequest so that partition leaders can validate Transaction Version 2 (TV2) end markers by requiring producerEpoch > currentEpoch instead of accepting equality. Without this, leaders could not distinguish late/duplicate TV1 markers from active TV2 transaction markers, creating a correctness gap in exactly-once semantics even after KIP-890 introduced epoch-bump semantics.

TLDR: Surfaces the rackId field already present in ConsumerGroupDescribeResponse and ShareGroupDescribeResponse into the MemberDescription and ShareMemberDescription objects returned by AdminClient. The rack ID was silently discarded during response parsing despite being available in the wire protocol, making it impossible to verify rack-aware assignment via the Admin API.

TLDR: Introduces share partition lag computation and persistence so that operators can query per-partition consumption progress for share groups via DescribeShareGroupOffsets and kafka-share-groups.sh. Share groups previously had no per-partition lag metric because their non-sequential, multi-consumer consumption model makes lag more complex than a simple end offset minus committed offset.

TLDR: Introduces an adaptive mode for group.coordinator.append.linger.ms (value -1) that dynamically adjusts batch linger time toward 0ms under low load and rises to reduce latency impact under high load. The fixed 5ms default linger introduced a minimum latency floor for all group coordinator and share coordinator writes under KIP-848's new coordinator runtime, causing latency regressions for low-throughput workloads after upgrading to Kafka 4.0.

TLDR: Adds an optional user (principal name) tag to the DeprecatedRequestsPerSec JMX metric, gated behind a new config kafka.metrics.deprecated.requests.tag.user.enabled. The existing metric from KIP-896 could not identify which principal was sending deprecated API requests in multi-tenant clusters where multiple users share the same client.

TLDR: Adds a lock renewal RPC for share consumers in explicit acknowledgement mode so that a consumer processing a long-running record can extend its acquisition lock timeout without changing the record's delivery state. Without renewal, a record under active processing would have its acquisition lock expire and become re-deliverable to another consumer if processing exceeded the lock timeout.

TLDR: Adds an application-id tag to the Kafka Streams client-state JMX metric so that multiple instances of the same logical Streams application can be grouped together in monitoring systems. The existing metric distinguished physical instances via a UUID processId but had no application-level grouping, making it impossible to correlate separate physical instances into a single logical application in dashboards.

TLDR: Extends kafka-broker-api-versions.sh to support the --bootstrap-controller flag, enabling operators to query the ApiVersions of KRaft controllers directly. Controllers have their own set of supported RPCs (e.g., metadata quorum RPCs) that are distinct from broker RPCs, but there was no tooling to inspect controller API capabilities independently of broker capabilities.

TLDR: Introduces two new KRaft configs — controller.quorum.fetch.max.bytes and controller.quorum.snapshot.max.bytes — to control the maximum bytes per Fetch and FetchSnapshot request respectively. New controllers joining a cluster with slow network links could get stuck cycling between Unattached/Follower/Prospective because the 8MiB default snapshot byte limit prevented completing a FetchSnapshot within fetch.timeout.ms, repeatedly triggering unnecessary elections.

TLDR: Updates the ClientTelemetryReceiver interface (introducing ClientTelemetryExporter and ClientTelemetryContext) to include the push interval in milliseconds alongside each client telemetry push. Without the push interval, MetricsReporter implementations receiving client telemetry could not determine when a client had disconnected versus simply not yet sent its next push, leaving stale metrics exposed indefinitely.

TLDR: Adds tasks-revoked-latency, tasks-assigned-latency, and tasks-lost-latency (avg + max) metrics for the Kafka Streams rebalance listener callbacks under the new KIP-848 streams rebalance protocol. Streams migrated from the consumer rebalance listener to a streams-specific implementation, dropping the latency metrics that previously existed for revoked/assigned/lost partition callbacks.

TLDR: Changes the type of `log.segment.bytes` from `int` to `long` to allow log segments larger than 2 GB. Modern high-capacity servers (e.g., EPYC nodes with 15 TB drives per disk) create disproportionately large numbers of segment files at the 2 GB limit, wasting file descriptors and increasing I/O overhead from frequent segment rotations.

TLDR: Deprecates the log.dir configuration (singular) in favor of the already-existing log.dirs (plural), since KIP-1161 made log.dir a LIST-type config accepting comma-separated paths, making both settings functionally identical. Having two configurations with the same behavior causes confusion for operators and pollutes the configuration surface.

TLDR: Adds a configuration flag to Kafka Connect workers to control whether internal topics (config, offsets, status) are automatically created if missing, defaulting to the current auto-create behavior. Auto-creation can corrupt an existing Connect cluster if a misconfigured worker starts against a wrong cluster or if partially deleted topics get silently recreated with wrong partition counts or replication factors.

TLDR: Adds a remote.log.metadata.admin. config prefix for TopicBasedRemoteLogMetadataManager (TRLMM) to pass admin client-specific configurations separately from producer (remote.log.metadata.producer.) and consumer (remote.log.metadata.consumer.) configs. Without this prefix, admin-specific settings (e.g., admin.request.timeout.ms) must be set via the common remote.log.metadata.common.client. prefix, which applies to all three client types and can cause unintended interactions.

TLDR: Fixes the RequestHandlerAvgIdlePercent JMX metric in KRaft combined mode (broker+controller co-located) where the value incorrectly reported ~2.0 instead of 0–1, and adds separate BrokerRequestHandlerAvgIdlePercent and ControllerRequestHandlerAvgIdlePercent metrics. In combined mode, two separate KafkaRequestHandlerPool objects both divided their idle time by numIoThreads before reporting to a shared Meter, effectively halving the denominator and doubling the reported ratio.

TLDR: Changes the maxFetchRecords limit in ShareFetch from a soft limit (batch-boundary aligned) to a strict record-count limit, ensuring the number of records returned never exceeds the configured value. The soft limit caused unpredictable batch sizes for share consumers with long per-record processing times, leading to frequent acquisition lock timeouts and uncontrolled redelivery.

TLDR: Extends the kafka-metadata-quorum describe command to display CommittedVoters—the set of controller voters that has been durably committed to the Raft log—separately from the in-memory VoterSet which may include uncommitted membership changes. During a controller membership change (add/remove voter), the in-memory and committed voter sets diverge; without surfacing CommittedVoters, operators cannot determine whether a membership change has been successfully committed to quorum.

TLDR: Allows operators to plug in a custom ReplicaPlacer implementation via a new unstable broker config, bypassing the default StripedReplicaPlacer round-robin algorithm. The default round-robin placement cannot be overridden without code changes, but some advanced use cases such as internal testing or specialized AZ-aware scenarios require custom placement strategies.

TLDR: Introduces configurable disk usage threshold configs (disk.max.used.percent, disk.warning.used.percent) so that brokers proactively reject new produce requests when disk usage exceeds the threshold, while remaining operational for reads, log cleanup, and admin operations. A disk-full condition currently causes a java.io.IOException that fails the log directory, offlines all partitions, and ultimately triggers a broker shutdown, requiring manual intervention to recover.

TLDR: Introduces a ConfigKey.Builder class with a fluent API as an alternative to the overloaded ConfigDef.define() methods for constructing ConfigKey instances. The growing number of parameters in ConfigDef.define() exceeded the practical limit where a builder pattern is recommended, and the lack of extensibility in ConfigKey made adding new functionality require widespread changes.

TLDR: Replaces the polling/retry initialization loop in TopicBasedRemoteLogMetadataManager with a direct dependency on broker readiness, adding a new initialization hook to avoid retrying before the broker can handle __remote_log_metadata requests. The default 2-minute retry timeout was insufficient for clusters that take more than 2 minutes to start, causing tiered storage initialization failures that left local log segments undeleted and produced excessive WARN logs.

TLDR: Introduces a group.coordinator.append.max.buffer.size config to cap the group coordinator's append cache buffer size independently of the broker's message.max.bytes setting. The current implementation derives the buffer ceiling from message.max.bytes, so deployments with large max message sizes (common in event pipelines) create excessive memory pressure on the group coordinator and risk OutOfMemoryError.

TLDR: Marks org.apache.kafka.streams.errors.BrokerNotFoundException as deprecated in Kafka 4.2 for removal in 5.0. The exception has been unused by Kafka Streams internals since version 2.8, leaving dead code that confuses users and complicates the exception class hierarchy.

TLDR: KIP-1194 changes the replica assignment algorithm to balance replica counts across individual brokers when rack configurations are uneven (i.e., racks have different numbers of brokers). The existing strategy prioritizes rack balance, which causes individual brokers in smaller racks to receive disproportionately more replicas and become hot spots.

TLDR: Deprecates MX4J support (the kafka_mx4jenable JVM flag) in Kafka 4.2 and removes it in 5.0. MX4J is a library for exposing JMX over HTTP that was last released in 2006, is completely undocumented in Kafka's official docs, and has had no reported users since a 2016 mailing list query, yet it adds maintenance burden and potential security exposure.

TLDR: Adds an --include argument to kafka-consumer-perf-test.sh (mutually exclusive with --topic) that accepts a regex pattern for multi-topic pattern subscription. The ConsumerPerformance tool only supported single-topic subscription while ConsoleConsumer already had --include, making it impossible to benchmark consumer performance across multiple topics or dynamic topic sets.

TLDR: Adds dead-letter queue (DLQ) support for share groups, automatically routing records that exceed their max delivery count or are explicitly rejected (ARCHIVED acknowledgment type) to a configurable dead-letter topic. KIP-932's share groups track delivery counts and enforce limits, but provided no automated mechanism to handle poison messages—they simply accumulated redeliveries until manually addressed.

TLDR: Adds an AvgIdleRatio metric to the ControllerEventManager showing the fraction of time the controller event loop thread is idle (0=always busy, 1=always idle). No metric existed for controller thread idleness, making it impossible to detect saturation, anticipate bottlenecks, or distinguish whether slow operations were caused by overloaded processing or simply rare events.

TLDR: Enables RemoteStorageManager (RSM) plugins to declare custom per-topic configuration properties and have the broker forward those topic-level configs to the RSM at runtime. The broker currently forwards only broker-level configs prefixed with rsm.config to the RSM, with no mechanism for RSM plugins to express or receive topic-level configuration overrides (e.g., different remote storage buckets per topic).

TLDR: Introduces a new built-in ConnectorClientConfigOverridePolicy named AllowlistConnectorClientConfigOverridePolicy with a configurable connector.client.config.override.policy.allowlist that restricts connector config overrides to an explicit set. Multiple CVEs (including RCE via SASL JAAS JndiLoginModule) exploited the default All policy; the None policy is too restrictive while the Principal policy proved unsafe.

TLDR: Extends DescribeLogDirs RPC and the kafka-log-dirs CLI to report remote (tiered) log size per partition in addition to local log size. KIP-405 introduced tiered storage, but remote segment metadata (size, offsets) is only used internally by the broker—administrators have no CLI tool to observe how much data is stored remotely, complicating capacity planning and tiered storage management.

TLDR: Fixes a deadlock in AddRaftVoterRequest handling by allowing the in-flight voter-add RPC to be superseded by new Fetch RPCs when the adding controller is also needed to commit the voter set change. The existing implementation required the controller being added to send a Fetch to commit the new voter set, but that controller's single in-flight slot was occupied by the pending AddRaftVoterRequest, causing the quorum leader to resign via checkQuorumTimer expiry.

TLDR: Proposes a unified log layer for Apache Kafka that supports both local replication-based storage and cloud shared storage (block, file, object) simultaneously, making Kafka stateless and removing the need for ISR replication when running on durable shared storage. Kafka's ISR replication was designed for local disks and duplicates durability guarantees already provided by cloud shared storage services, wasting resources and preventing stateless elasticity.

TLDR: Proposes a Quality of Service (QoS) framework for Kafka that allows producers, brokers, and consumers to declare desired performance characteristics (latency SLA, throughput targets) and for the cluster to expose its actual QoS capabilities and observations. Kafka and its compatible implementations vary significantly in performance and reliability characteristics with no standard mechanism for clients to negotiate or discover these constraints.

TLDR: Adds generic feature-level JMX metrics (FinalizedLevel, minimum-supported-level, maximum-supported-level) tagged by feature name for each production feature (metadata.version, kraft.version, transaction.version, group.version, etc.). Only metadata.version had a dedicated metric; as new features were added (ELR version, share version, streams version), there was no way to monitor their finalized or supported levels for safe upgrade/downgrade planning.

TLDR: Introduces a new consumer config remote.max.partition.fetch.bytes that controls the maximum bytes per partition returned from tiered remote storage in a Fetch request, propagated to the broker via a new FetchRequest v18 field. The existing max.partition.fetch.bytes applied to both local and remote reads uniformly, but remote storage plugins are often tuned for larger read chunks (e.g., 4MB), and reading only 1MB per remote fetch increased round-trip overhead without benefiting local read performance.

TLDR: Proposed extending tiered storage to upload the active (open) log segment to object storage so that follower replicas could replicate from object storage rather than from the leader broker. This KIP was withdrawn; the use case is covered by the Diskless Topics design (KIP-1150) which takes a more fundamental approach to eliminating local disk replication.

TLDR: Deprecates the misspelled ProducerConfig constant PARTITIONER_ADPATIVE_PARTITIONING_ENABLE_CONFIG and introduces the correctly-spelled PARTITIONER_ADAPTIVE_PARTITIONING_ENABLE_CONFIG alias for removal in the next major version. The typo in the Java constant name made documentation, code search, and autocomplete unreliable for users configuring adaptive partitioning.

TLDR: Proposes allowing multiple Kafka Connect clusters to share a single set of internal storage topics (offset, status, config) rather than requiring three dedicated topics per cluster. Each new Connect cluster requires three compacted topics, leading to topic proliferation, cross-team provisioning bottlenecks, and excessive compaction overhead as deployment scale grows.

TLDR: Propagates the follower's current high-watermark (HWM) in the KRaft Fetch request so the leader can immediately complete parked Fetch requests when only the HWM has advanced (without new records). KRaft could not distinguish whether a follower was behind on records versus behind on HWM, so it unconditionally parked Fetch requests for up to 500ms even when only the HWM needed propagation, causing admin operation metadata updates to appear delayed by up to 500ms.

TLDR: Describes the Diskless Coordinator, a new component that manages per-partition metadata (batch locations, WAL file references, producer state, transaction state) for Diskless Topics on behalf of brokers. Because Diskless Topics bypass local disk and direct replication, brokers need an authoritative external metadata service to answer offset queries, commit batch writes, and coordinate truncation.

TLDR: Specifies the produce and consume paths for Diskless Topics (introduced by KIP-1150), where records are written to shared object storage via a Write-Ahead Log (WAL) without appending to local block devices or performing direct ISR replication. The existing classic topic architecture requires durable local block storage and peer-to-peer replication, making it unsuitable for cloud environments where object storage is cheaper and more elastic.

TLDR: Standardizes LIST-type configuration validation across Kafka to disallow empty lists and duplicate values in configs where such inputs cause misconfiguration or silent semantic redundancy. Previously, LIST-type configs accepted empty or null values without error and allowed duplicates without warning, misleading operators who expected validation at configuration load time.

TLDR: Adds an optional nodeId parameter to the DescribeFeaturesOptions in AdminClient and a --node-id argument to the kafka-features.sh describe command so that supported feature versions can be queried from a specific node. Supported feature levels depend on the unstable.api.versions.enable flag which can differ per node, but AdminClient previously sent the request to an arbitrary node, making it impossible to check a specific node's supported range.

TLDR: Proposes a reference-based Serializer/Deserializer that transparently stores large payloads in external object storage (e.g., S3) and embeds only the storage reference in the Kafka message. Kafka's hard limit on message.max.bytes cannot be raised indefinitely without causing broker memory, network, and stability issues, and existing workarounds (chunking, manual reference patterns) require complex application-level logic.

TLDR: Makes KafkaPrincipalBuilder extend KafkaPrincipalSerde, converting the need for custom principals to implement serialization into a compile-time requirement rather than a runtime failure. KRaft requires brokers to forward requests to controllers with serialized principal objects; without KafkaPrincipalSerde, custom KafkaPrincipalBuilder implementations silently failed at runtime during request forwarding.

TLDR: Introduces metadata version downgrade support, allowing a KRaft cluster to roll back its MetadataVersion to an earlier value after a failed or undesirable upgrade. Currently, MetadataVersion is a one-way ratchet—clusters can upgrade but never downgrade—meaning operators who encounter bugs in a new metadata version have no recovery path short of a full cluster rebuild.

TLDR: Extends Kafka Connect's timestamp logical type support to include microsecond (`timestamp-micros`) and nanosecond (`timestamp-nanos`) precision alongside the existing millisecond-only `Timestamp` type. Source systems and formats such as Avro and Parquet support sub-millisecond timestamp precision; Connect's millisecond truncation silently loses precision when ingesting or egressing high-resolution timestamp data.

TLDR: Deprecates KafkaStreams.CloseOptions (inner class with public constructor) and replaces it with a new top-level CloseOptions class using a fluent builder API, aligned with KIP-1092's CloseOptions for the consumer. The existing CloseOptions exposed a public constructor rather than following the fluent API style used consistently throughout Kafka Streams configuration objects.

TLDR: Adds a `TransactionalIdPattern` (RE2J regex) filter to the `ListTransactions` admin API so clients can filter transactions by transactional ID pattern server-side alongside the existing state, producer ID, and duration filters. On large clusters with millions of transactional IDs, fetching all transactions and filtering client-side wastes network bandwidth and broker CPU.

TLDR: Proposes a minimal-movement replica balancing algorithm for `kafka-reassign-partitions.sh` that computes an assignment minimizing the number of replica relocations while still achieving even distribution across brokers. The current `kafka-reassign-partitions.sh --generate` algorithm does not consider the existing replica placement, producing assignments that move far more replicas than necessary and causing excessive bandwidth consumption during rebalancing.

TLDR: Introduces Diskless Topics, a new topic type that stores all data in shared object storage (e.g., S3) without writing to local broker disk or performing ISR-based replication. Classic Kafka topics require expensive replicated block storage for active segments even in cloud environments where high-durability object storage is far more cost-effective, blocking truly elastic, serverless Kafka deployments.

TLDR: Proposes an official Apache Kafka Helm chart for Kubernetes deployments using the Apache-published Docker image, with all broker properties templated via Helm values and stored as ConfigMaps. Existing Kubernetes deployment options (Bitnami Helm chart, Confluent/Strimzi operators) use proprietary container images and require additional managed-service components, while no official lightweight chart existed using the upstream Apache image.

TLDR: Deprecates `log.cleaner.enable` and removes it in Kafka 5.0, and sets a lower bound of 1 on `log.cleaner.threads`. The configuration is effectively a no-op since any compacted topic requires an active log cleaner; keeping it creates misleading silent failures (compaction silently stops) and unnecessary null-check complexity throughout the `LogCleaner`/`LogManager` code paths.

TLDR: Adds --bootstrap-server to kafka-producer-perf-test.sh, introduces --command-property as a consistent replacement for --producer-props, and adds --command-property to kafka-consumer-perf-test.sh and kafka-share-consumer-perf-test.sh. Different perf tools used different argument names for the same concepts (e.g., bootstrap.servers vs --bootstrap-server), creating unnecessary friction and inconsistency for operators running comparative benchmarks.

TLDR: Adds anchored (wall-clock-aligned) punctuation to Kafka Streams so that periodic callbacks can be triggered at fixed calendar intervals (e.g., every hour at HH:00:00) regardless of when the punctuation was registered. Existing wall-clock punctuation starts its interval from registration time, producing non-deterministic trigger offsets relative to wall clock, which caused complex workarounds for time-aligned processing in sectors like energy grid balancing.

TLDR: Introduces a dedicated REST API layer for MirrorMaker2 that abstracts over the Connect REST API, exposing replication-flow–centric endpoints (flows, connectors, tasks, replicated topics) rather than generic connector endpoints. Operators currently must use low-level Connect REST endpoints or parse logs to inspect MirrorMaker2 state; the lack of a purpose-built API makes automation and monitoring difficult.

TLDR: Extends `kafka-configs.sh --describe --all-groups` to include groups that have dynamic configuration but no active members or committed offsets. Currently, listing all groups' dynamic configs is impossible because `--all-groups` only surfaces groups known to the group coordinator, excluding groups that exist solely via config records.

TLDR: Changes `KafkaConsumer.offsetsForTimes()` to return `Optional<OffsetAndTimestamp>` instead of `OffsetAndTimestamp` (nullable) for partitions with no matching timestamp, removing null values from the returned map. Returning null values in a `Map` violates the principle of least surprise and causes `NullPointerException` when callers iterate the map without null checks.

TLDR: Adds support for the OAuth 2.0 `urn:ietf:params:oauth:grant-type:jwt-bearer` grant type (RFC 7523) in Kafka clients alongside the existing `client_credentials` grant, allowing authentication via a signed JWT assertion instead of a plaintext secret. The `client_credentials` grant requires embedding plaintext secrets in client configuration, which many organizations and cloud providers prohibit for security compliance.

TLDR: Cleans up the Kafka Streams configuration API by deprecating TopologyConfig and requiring topology-specific configs (topology.optimization, processor.wrapper.class, etc.) to be passed via StreamsConfig directly to both KafkaStreams and the topology builder. Topology-specific configs passed only to KafkaStreams were silently ignored at the topology level, and multiple partially-overlapping config APIs (TopologyConfig, StreamsBuilder constructor, Topology constructor) caused fragmentation and misconfiguration.

TLDR: Establishes a uniform configuration precedence order (CLI args > CLI properties > config files > defaults) across all Kafka tool scripts, with a new --modern flag for scripts that don't yet follow it. Inconsistent precedence across tools caused user confusion and made configurations from files or properties silently ignored when multiple sources were present.

TLDR: Makes ConsumerGroupMetadata an interface instead of a concrete class so that frameworks (e.g., Kafka Streams with EOS) can provide custom implementations without depending on the internal constructor. The class is passed to KafkaProducer.sendOffsetsToTransaction() to carry group metadata for transactional offset commits; making it an interface enables mock implementations in tests and custom metadata carriers in frameworks.

TLDR: Introduces Virtual Clusters as a multi-tenancy layer on top of a shared Kafka cluster, providing each tenant with isolated topic namespaces, quotas, and ACLs through a proxy or broker-side virtualization layer. Operators running multi-tenant Kafka must today either provision separate physical clusters per tenant (expensive, wasteful) or manually partition namespaces with naming conventions and ACLs (error-prone, no true isolation).

TLDR: Migrates the Apache Kafka documentation website from raw HTML with Handlebars.js templates and server-side includes to Markdown with a Hugo/Docsy static site generator. The existing raw HTML documentation mixed content with styling, had inconsistent heading levels, required a web server for testing, and was a high barrier to contribution for developers unfamiliar with HTML/CSS.

TLDR: KIP-1132 fixes KRaft brokers to tolerate static `SocketServer` configurations that would be invalid in isolation (e.g., a listener referencing a certificate not yet present on disk) when a prior dynamic reconfiguration already makes the effective configuration valid. Brokers crashed on startup with an uncaught exception whenever the static config was syntactically invalid, even if the dynamic config override rendered it harmless.

TLDR: Adds per-broker `BrokerRegistrationState` metrics and a `ControlledShutdownBrokerCount` metric on the KRaft controller to expose individual broker states without requiring log inspection. The existing `ActiveControllerCount` and fenced/unfenced broker count metrics provide only aggregate totals; diagnosing which specific broker entered a particular state requires manual parsing of the metadata log.

TLDR: Adds two new broker metrics — `waiting-connection` (gauge of connections queued for an available slot) and `connection-latency` (histogram of connection wait time) — per listener at the Acceptor level. When clients hit broker connection limits or throttling, they receive only a timeout exception with no broker-side visibility into how many connections are backlogged or how long they wait.

TLDR: Deprecates KTable.transformValues (still using the old Processor API) and replaces it with KTable.processValues using the new typed Processor API; also deprecates KStreams.processValues to fix a backward-incompatibility bug introduced in its implementation. The old Processor API classes could not be fully removed in Kafka 4.0 because KTable.transformValues still referenced them, and a bug in KStream.processValues made migration via the new API unsafe.

TLDR: Adds a new `FlexibleWindows` window type to Kafka Streams that retains open windows indefinitely until explicitly closed by a grace period expiry or new records, rather than discarding late-arriving records that fall outside a fixed window end time. Existing Kafka Streams window types (tumbling, hopping, sliding, session) all rely on event-time boundaries and discard truly late data, which breaks use cases such as micro-batching by offset order where no record should ever be dropped.

TLDR: Adopts GitHub Merge Queues to serialize all changes to Kafka's mainline branches so each commit is validated against the current HEAD before merging. As the contributor rate increases, concurrent PR merges without serialization allow individual green PRs to have bad interactions that break the main branch when combined.

TLDR: Removes the `opennumberfiles` metric from the RocksDB state store in Kafka Streams, which always returns -1 since RocksDB 8.7.3 removed the underlying `NO_FILE_CLOSES` internal counter. The stale metric misleads operators and can trigger false alarms in monitoring systems.

TLDR: Adds rack-aware (AZ-aware) partitioning to the Kafka producer: when client.rack is set, the producer preferentially routes to partitions whose leaders reside in the same AZ, reducing cross-AZ network traffic and latency. The default adaptive partitioner rotated across all partitions without rack awareness, causing producers to regularly send records cross-AZ and incur inter-AZ data transfer costs in cloud deployments.

TLDR: Extracts the Kafka Connect data format classes (Schema, Struct, SchemaBuilder, etc.) from the `connect-api` module into a standalone `connect-data` module. This removes the coupling that forces non-Connect projects to depend on the entire Connect API just to use Kafka's structured data model.

TLDR: Proposes a hardware acceleration framework for Kafka's existing compression algorithms (gzip, zstd, snappy, lz4), allowing producers, brokers, and consumers to offload compression/decompression to hardware accelerators (e.g., Intel QAT, GPU) without changing the wire format. Compression improves throughput and reduces storage costs but adds CPU latency; hardware accelerators can reduce that latency without sacrificing compression ratio, but Kafka has no pluggable acceleration abstraction.

TLDR: Fixes AppInfo metrics registration for Kafka Connect workers and MirrorMaker 2 clients to include the client-id label, aligning them with the behavior of standalone producer, consumer, and admin clients. AppInfo metrics (start-time-ms, commit-id, version) lack client-id tags for worker and MM2 contexts, making it impossible to distinguish metrics from multiple workers or MM2 instances sharing the same JVM.

TLDR: Adds `ssl.auto.reload` configuration to Kafka clients (producers, consumers) and brokers so SSL/TLS certificates are automatically reloaded when updated on disk without requiring a restart. Currently, only brokers support dynamic SSL certificate rotation via dynamic config; producers and consumers require disruptive restarts when certificates are rotated by external agents.

TLDR: Adds deadlock detection to `KafkaProducer` so that calling `Producer#flush()` from within a `send()` callback throws `KafkaException` instead of blocking indefinitely. Invoking `flush()` from the I/O thread that executes send callbacks creates an irresolvable circular wait, causing application hangs that are difficult to diagnose.

TLDR: Adds a new ssl.keystore.alias config that instructs the DefaultSslEngineFactory to select a specific key alias from a multi-entry keystore when constructing the SSLContext, rather than defaulting to the first entry. Keystores with multiple key entries caused SSL handshake failures or wrong-certificate authentication because Kafka had no mechanism to select a specific alias.

TLDR: KIP-1116 proposes adding new principal types beyond `User` to the standard ACL system so that ACL rules can match groups of `KafkaPrincipal` identities without losing the original client identity in logs. Current workarounds using principal mapping rules (to embed group membership in the principal name) discard the unique client identity, making audit log attribution impossible.

TLDR: Proposes migrating Apache Kafka's build system from Gradle to Bazel, leveraging Bazel's fine-grained build graph, aggressive caching, and hermetic toolchains. Gradle builds had non-deterministic dependency graphs (no lockfile), per-module dependency resolution (risking version conflicts), and CI times of ~2 hours average, while internal Confluent Bazel builds achieved ~30-minute average CI times.

TLDR: Proposes splitting a Kafka partition's data into immutable inactive chunks and a single active chunk, allowing inactive chunks to be reassigned to different brokers independently of the active chunk. The entire partition replica is managed as one unit for assignment and rebalancing, making partition moves expensive even when only the active (tail) data matters for most operations.

TLDR: Adds a `ProcessorWrapper` interface to Kafka Streams (`StreamsConfig.PROCESSOR_WRAPPER_CLASS_CONFIG`) that allows users to wrap every processor in a topology with a common decorator (logging, tracing, error handling) without modifying each processor individually. Applying cross-cutting concerns to every processor currently requires manual wrapping at every PAPI processor and is completely impossible for DSL-generated processors.

TLDR: Adds a validation mode to Kafka Streams that enforces explicit naming of all internal repartition and changelog topics, failing topology construction when auto-generated names are used. Auto-generated names embed topology node indices, so any topology restructuring silently renames internal topics and causes state loss or repartition mismatches.

TLDR: Standardizes Kafka consumer topic-level metric names to use the original topic name (preserving dots) instead of replacing dots with underscores, aligning consumer metric naming with the existing producer convention. The current asymmetry between producer metrics (dots preserved) and consumer metrics (dots replaced by underscores) causes confusion and breaks monitoring dashboards that use unified naming.

TLDR: Allows producers to override `acks` and `compression.type` on a per-topic basis rather than only at the producer level. Since `KafkaProducer` is designed to be shared across threads writing to different topics, there is no current mechanism to send to one topic with `acks=all` and another with `acks=1` from the same producer instance.

TLDR: Adds a duration-based offset reset option (`by-duration`) to consumer clients and the `kafka-consumer-groups.sh` tool, allowing consumers to reset to the offset corresponding to `now - duration` rather than only to earliest/latest/specific timestamp. With tiered storage enabling long retention, operators need a human-friendly way to seek back by time interval without computing an absolute timestamp.

TLDR: Makes the `RemoteLogManager` thread pool sizes (`remote.log.manager.copier.thread.pool.size`, `remote.log.manager.expiration.thread.pool.size`, `remote.log.reader.threads`) dynamically reconfigurable without broker restart. Under variable tiered storage load, operators need to tune concurrency at runtime to avoid backlogs or over-provisioning.

TLDR: Adds a BiFunction<K, V, KO> foreignKeyExtractor overload to KTable.join() and related DSL methods so that the foreign key can be derived from both the record key and value, not just the value. The existing Function<V, KO> extractor forced users to duplicate key fields into the record value whenever the foreign key depended on the message key, creating data redundancy and potential inconsistency.

TLDR: Adds broker-side Share Group metrics: TotalShareFetchRequestsPerSec, FailedShareFetchRequestsPerSec, TotalShareAcknowledgementRequestsPerSec, and related per-topic and purgatory metrics. Share Group (KIP-932) cooperative consumption had no dedicated broker metrics for share fetch requests, acknowledgements, or Delayed Share Fetch purgatory behavior, preventing observability and performance diagnosis.

TLDR: Extends client rebootstrap (KIP-899) to trigger not only when all metadata nodes are unreachable, but also after configurable timeouts or on specific broker error codes indicating stale metadata. This covers edge cases such as rolling broker replacements where old IPs become unresponsive but no connection error is immediately returned.

TLDR: Replaces per-partition rack topology data (the full set of replica racks) in the ConsumerGroup subscribed topic metadata with a compact hash, and triggers rebalances when the hash changes due to partition additions or rack changes. Storing the full rack set per partition for groups with thousands of members and thousands of partitions consumed disproportionate memory (up to 79% of ConsumerGroup object size in production cases) after the new consumer group protocol (KIP-848) was introduced.

TLDR: Renames the incorrectly prefixed metrics org.apache.kafka.server:type=AssignmentsManager and org.apache.kafka.storage.internals.log:type=RemoteStorageThreadPool back to the canonical kafka.server:type=AssignmentsManager and kafka.log.remote:type=RemoteStorageThreadPool naming convention. A class relocation and a patch introduced non-standard package-based JMX bean prefixes that diverged from Kafka's kafka.COMPONENT convention, breaking monitoring dashboards.

TLDR: Extends ConsumerGroupDescription with optional groupEpoch and targetAssignmentEpoch fields, MemberDescription with memberEpoch and upgraded fields, and ShareGroupDescription with groupEpoch and targetAssignmentEpoch, surfacing these in kafka-consumer-groups.sh and kafka-share-groups.sh. The new consumer group protocol (KIP-848) introduced epoch-based assignment tracking but the admin tooling had no visibility into group or member epochs.

TLDR: Extends MirrorMaker2 checkpointing to support reverse replication so that consumer group offsets made on the downstream cluster are checkpointed back upstream, enabling minimal reprocessing on failback after a failover. In bidirectional replication, the standard checkpoint flow only covers failover (upstream→downstream direction), leaving consumers that progressed on the downstream cluster unable to resume accurately when switching back.

TLDR: Proposes a pluggable ErrorHandler interface for Kafka Connect that allows connector developers to define custom per-record error handling logic (DROP, FAIL, ACK) during specific processing stages. Connect's existing error tolerance (none/all) and dead letter queue were coarse-grained, with no way to implement stage-specific or record-specific handling logic comparable to Kafka Streams' ProcessingExceptionHandler.

TLDR: Proposes a Kafka Canary service that sends small probe messages through the full produce/consume path in isolation from production traffic, enabling SLA monitoring and early detection of deployment-induced failures. Even with thorough testing, bad deployments (misconfiguration, untested scenarios) can cause silent data loss or latency regressions that only manifest in production, requiring an automated canary layer separate from business traffic.

TLDR: Adds a `nextOffsets` map to `ConsumerRecords` that exposes, per partition, the offset to be fetched in the next `poll()` call (i.e., `lastOffset + 1`), pre-wrapped in `OffsetAndMetadata`. This allows Kafka Streams and other frameworks to commit precise next-fetch offsets without re-computing them from the last received record.

TLDR: Adds high-level MirrorMaker2 metrics for `topic-count`, `partition-count` per topic, `consumer-group-count`, and `consumer-group-offset` to complement the existing record-level metrics. The current MM2 metrics cover data throughput and replication latency but provide no visibility into the aggregate number of topics and consumer groups being synchronized, making capacity planning and correctness verification opaque.

TLDR: Introduces numeric JMX metrics for StreamThread state (CREATED/RUNNING/DEAD etc.) and KafkaStreams client state, plus a metrics.recording.level metric, enabling operator monitoring systems that require numeric gauges rather than string values. Existing Kafka Streams state metrics were string-typed and incompatible with KIP-714/KIP-1076 telemetry pipelines, making automated health alerting difficult.

TLDR: Introduces a formal flaky test management process including a `@Flaky` annotation to tag known-flaky tests, a threshold mechanism to quarantine tests exceeding a flakiness rate, and a dedicated re-run strategy for annotated tests. Unmanaged flaky tests erode CI confidence, mask real failures, and waste infrastructure by requiring full suite re-runs up to three times to achieve reliable results.

TLDR: Adds a `replication.heartbeats.topic.enabled` configuration option to `MirrorSourceConnector` to allow users to opt out of replicating the heartbeat topic between clusters. When running multiple `MirrorSourceConnector` instances replicating different topic subsets between the same two clusters, heartbeat topic replication cannot be scoped to a single connector and creates unwanted topic duplication.

TLDR: Replaces the `KafkaClientSupplier` interface in Kafka Streams with a new `KafkaClientInterceptor` that wraps clients after construction rather than supplying them whole, preserving the runtime's ability to build clients using internal APIs. As Kafka Streams moves toward using internal consumer primitives (KIP-1071) it can no longer treat client creation as a black-box user concern without losing the ability to co-develop clients and Streams internals.

TLDR: Deprecates and removes the `--intermediate-topics` option from the `StreamsResetter` tool in version 5.0. The `StreamsResetter`'s `intermediateTopicsOption` was only relevant for topologies using the `KStream#through()` method, which was deprecated and subsequently removed; the option is now a dead code path.

TLDR: Deprecates several `*_DOC` constants in `StreamsConfig` that have public visibility but are only used internally by `StreamsConfig` or `TopologyConfig`, with a plan to make them private in the next major release. Leaking internal documentation string constants as public API members unnecessarily enlarges the public surface area and risks accidental external usage that would prevent future refactoring.

TLDR: KIP-1083 increases the default value of `task.shutdown.graceful.timeout.ms` in Kafka Connect from 5 seconds to a larger value. On large Connect workers running many tasks, 5 seconds is insufficient for all tasks to flush and commit their state before being forcibly terminated, leading to unnecessary reprocessing or data loss on restart.

TLDR: Requires clients to generate and send member IDs on the initial `ConsumerGroupHeartbeat` RPC rather than letting the broker assign them, making the first heartbeat idempotent. Without client-generated IDs, a consumer that sends a join heartbeat but closes before receiving the response may issue a leave heartbeat with a null member ID, causing the broker to register an orphaned member that never expires.

TLDR: Deprecates the misspelled `maxlifeTimeMs()` method in `CreateDelegationTokenOptions` and adds a correctly named `maxLifetimeMs()` replacement. The typo (`maxlifeTimeMs` vs. `maxLifetimeMs`) has been in the public API since delegation token creation was introduced, creating a poor developer experience and a breaking rename burden if left uncorrected past the next major release.

TLDR: Deprecates the `--delete-config` option of `kafka-topics.sh` (`TopicCommand`) with a plan to remove it in Kafka 5.0. The `--delete-config` command is currently a no-op (it performs no actual deletion), so retaining it without deprecation misleads users into believing topic-level config keys can be removed via this path.

TLDR: Deprecates gracePeriod(), keySerde(), valueSerde(), and otherValueSerde() getter methods from the public Joined helper class, since these methods expose internal concern and are already available on the internal JoinedInternal class used by the DSL implementation. These getters leaked implementation details into the public API, and removing them prevents external code from depending on internals.

TLDR: KIP-1077 deprecates `org.apache.kafka.streams.kstream.ForeachProcessor` and moves it to the internal package `org.apache.kafka.streams.kstream.internals`. The class was mistakenly placed in a public package despite being an internal implementation detail, preventing API evolution without going through the KIP process.

TLDR: Extends KIP-714's client metrics telemetry framework to include application-level metrics from Kafka Streams instances, not just the embedded producer/consumer clients. Cluster operators can observe client-side metrics via the broker pull mechanism, but Kafka Streams applications emit additional metrics (task-level lag, state store sizes, etc.) that remain invisible without this extension.

TLDR: KIP-1075 introduces a delayed purgatory (`DelayedRemoteListOffsets`) and a new `TimeoutMs` field in `ListOffsetsRequest` so timestamp-based offset lookups that require fetching remote indexes are handled asynchronously by dedicated remote-log-reader threads rather than blocking request-handler threads. `ListOffsets` requests against remote-storage-enabled topics require non-deterministic latency to fetch remote offset and time indexes; when many such requests arrive concurrently, request-handler thread exhaustion starves higher-priority `FETCH` and `PRODUCE` requests.

TLDR: Adds a `replication.policy.internal.topic.separator.enabled` configuration (and related `ReplicationPolicy` changes) so MirrorMaker2 can replicate user-defined topics whose names end in `.internal` or `-internal`. The current `ReplicationPolicy.isInternalTopic()` logic unconditionally excludes any topic matching the internal naming pattern, silently dropping replication of legitimate business topics with those suffixes.

TLDR: Extends `DescribeClusterRequest` (v2+) to optionally include fenced brokers in the response, adding an `includeFencedBrokers` option to `DescribeClusterOptions` and a `Fenced` field per broker in the response. When a KRaft broker node is removed without being unregistered, it stays fenced in the controller but is invisible to `DescribeCluster`, preventing operators from discovering the node ID needed to call the unregister API.

TLDR: Introduces a native Streams Rebalance Protocol that moves task assignment from a client-side custom PartitionAssignor piggyback to a server-side broker protocol built on the KIP-848 group coordinator. The existing approach serializes Streams task assignment state into the consumer group join/sync protocol payload, coupling Streams scheduling correctness to the slow, stop-the-world Classic rebalance and preventing incremental assignment.

TLDR: Deprecates MockProcessorContext (old test API), Transformer, TransformerSupplier, ValueTransformer, and ValueTransformerSupplier to complete the removal of the old Kafka Streams Processor API. These interfaces were already functionally replaced by their new Processor API equivalents (api.MockProcessorContext, FixedKeyProcessor, etc.) but could not be removed without first being deprecated.

TLDR: Adds new metrics specific to the `AsyncKafkaConsumer` (CONSUMER rebalance protocol) including thread coordination metrics and event queue depth that are not applicable to the `LegacyKafkaConsumer` (CLASSIC protocol). The new consumer implementation introduced in KIP-848 uses a dual-thread model (API thread + network I/O background thread) whose operational characteristics cannot be observed with the existing single-threaded consumer metrics.

TLDR: Adds a cordoning mechanism for brokers and log directories that marks them as unavailable for new partition placement while still allowing existing partitions to remain. The round-robin partition placer had no concept of planned maintenance, making broker decommissioning or JBOD disk removal painful because newly created partitions would still be assigned to the target being drained.

TLDR: Upgrades Kafka's logging interface from SLF4J 1.7 to SLF4J 2.0 and updates the default log4j2 binding to log4j-slf4j2-impl, enabling the -Dslf4j.provider JVM property for logging backend selection. SLF4J 1.x required classpath ordering tricks or manual JAR removal to select a logging backend, and Kafka's dependency on it prevented users from easily switching to modern logging implementations.

TLDR: Extends KIP-966's pagination pattern to additional admin protocol requests—OffsetFetchRequest and DescribeLogDirsRequest—by adding a max-partitions limit and a cursor field for continuation. On large clusters with thousands of partitions, these requests can return responses that exceed broker timeout limits, causing failures in admin operations.

TLDR: KIP-1061 proposes adding a `--describe` export mode to `kafka-configs.sh` that outputs SCRAM credential fields (`salt`, `stored_key`, `server_key`) needed to replicate credentials to another cluster. KIP-554 stored SCRAM credentials in KRaft metadata but intentionally withheld these fields from the describe output; without them, migrating a cluster requires all users to reset their passwords in the new cluster.

TLDR: KIP-1058 adds a `txnIdxEmpty` tagged field to `RemoteLogSegmentMetadataRecord` and `RemoteLogSegmentMetadataSnapshot` so the Remote Log Manager can skip fetching remote transaction indexes for segments that have no aborted transactions. When `isolation.level=READ_COMMITTED` consumers fetch from topics with no aborted transactions, the broker currently scans all remote segment transaction indexes sequentially before reaching local segments, causing excessive remote storage API calls and increased fetch latency.

TLDR: Adds a --remote-log-metadata-decoder flag to kafka-dump-log.sh that decodes RemoteLogSegmentMetadata records stored in the __remote_log_metadata internal topic using RemoteLogMetadataSerde. Debugging tiered storage issues required manual decoding of opaque binary records in this topic, since dump-log only supported decoders for __consumer_offsets, __transaction_state, and other internal topics.

TLDR: Deprecates default.deserialization.exception.handler and default.production.exception.handler StreamsConfig keys and adds replacement names without the default. prefix (deserialization.exception.handler, production.exception.handler). The default. prefix implies that per-topic overrides are possible, but these handlers have no per-topic override mechanism, making the prefix semantically incorrect.

TLDR: Adds a round-robin element distribution method to ConnectorUtils in Kafka Connect, assigning elements to groups in interleaved order (e.g., {a,d}, {b,e}, {c}) rather than the existing contiguous grouping ({a,b}, {c,d}, {e}). Connector workload distribution via contiguous grouping produces uneven load when groups must process each element's work serially, since the last group may receive fewer or lighter elements.

TLDR: Introduces an `external.schema` configuration option for `JsonConverter` in Kafka Connect that reads the schema from a file path rather than embedding it in every message payload. When connectors require a schema (e.g., JDBC connectors) but the message producer cannot embed a schema in each JSON record, the current approach either bloats every message with a full schema copy or requires a schema registry.

TLDR: Proposes renaming constants in CommonClientConfigs and StreamsConfig from a DEFAULT_VARIABLE_NAME pattern to a VARIABLE_NAME_DEFAULT suffix pattern to align with standard Java naming conventions. The DEFAULT_ prefix convention was inconsistent with how most Java frameworks represent default values, making the API harder to navigate.

TLDR: Adds --warmup-secs and --warmup-records options to kafka-producer-perf-test.sh to exclude the high-latency initialization phase from steady-state measurements. Without a warmup phase, performance test results included startup latency spikes from producer initialization and ISR establishment, requiring multi-hour test runs to dilute the startup noise below 1% of the measurement window.

TLDR: Allows `leader.replication.throttled.rate` and `follower.replication.throttled.rate` to be configured statically in broker properties files in addition to the existing dynamic-only support. Dynamic-only throttle configuration cannot protect against unplanned replication spikes (e.g., a broker rejoining after extended downtime) since an operator must react after the spike begins rather than proactively setting a safe ceiling.

TLDR: Defines a comprehensive error taxonomy for Kafka transactional and idempotent producer APIs — classifying each error as retriable, abortable, or fatal — and aligns error handling behavior across all client SDK implementations (Java, librdkafka, Go, .NET, Python). Inconsistent error handling across SDKs and incomplete error specifications in KAFKA-7787/KAFKA-14439 made it difficult for non-Java clients to implement correct transactional semantics.

TLDR: KIP-1049 adds the `log.summary.interval.ms` configuration to Kafka Streams, controlling how frequently thread-level summary statistics are logged (default 2 minutes). The hardcoded 2-minute interval cannot be adjusted, causing log pollution in busy production environments and insufficient visibility in debugging scenarios.

TLDR: Extends kafka-consumer-perf-test to support consuming from a single specified partition via a new --partition flag, enabling partition-level throughput and latency benchmarking. The tool previously subscribed to all partitions, making it impossible to isolate per-broker or per-partition performance and diagnose hardware-level issues (e.g., a faulty switch affecting one broker).

TLDR: Promotes `producer.id.expiration.check.interval.ms` from an internal static config to a dynamic public broker configuration. Without this, operators cannot tune the frequency of producer ID expiration checks at runtime, making `producer.id.expiration.ms` dynamic-only in name since the check interval stays fixed.

TLDR: KIP-1045 proposes moving `MockAdminClient` from the internal test source tree to the public API alongside `MockConsumer` and `MockProducer`. Developers testing applications that use `AdminClient` have no supported mock implementation, forcing them to either stand up a real broker in unit tests or maintain their own incomplete mock.

TLDR: Proposes changes to the idempotent producer's server-side PID (producer ID) tracking to address OOM failures caused by an unbounded explosion in the number of tracked PID/epoch pairs and to reduce the window between when a PID is received and when it is durably persisted. The fixed-size circular buffer design in the original KIP-98 spec was not implemented as specified, leading to unbounded memory growth in clusters with many concurrent transactional producers.

TLDR: Adds unified group administration APIs and tooling (AdminClient and CLI) that work consistently across all group types (classic consumer, new consumer, share, streams, Connect worker groups) using a single DescribeGroups-style RPC that identifies group type. The existing tools and APIs were group-type-specific: kafka-consumer-groups.sh returned misleading errors for share groups, and there was no unified way to list or describe groups across all types.

TLDR: Adds a pluggable `ProducerErrorHandler` interface that users can configure on `KafkaProducer` to intercept and handle producer-level errors (serialization, send failures) with custom logic. Currently producers only support fixed retry/fail semantics with no user hook for custom error routing or dead-letter handling.

TLDR: Allows the WriteTxnMarkers API to be authorized with the Alter permission on the Cluster resource in addition to the existing ClusterAction permission, enabling non-broker admin clients to abort hanging transactions (as introduced by KIP-664) without needing ClusterAction. ClusterAction is reserved for inter-broker communication and granting it to operator tooling violates the principle of least privilege.

TLDR: Extends `RecordDeserializationException` to carry the raw key bytes, value bytes, headers, timestamp, and timestamp type of the failed record, plus a `DeserializationExceptionOrigin` enum indicating whether the key or value caused the failure. KIP-334 added offset information to this exception for skip-and-continue semantics, but without access to the raw record bytes, implementing a dead-letter queue or diagnostic logging of poison-pill records requires a separate consumer re-fetch.

TLDR: Allows Kafka Streams StateStore implementations to manage their own changelog topic offsets rather than delegating that responsibility to the engine's per-task .checkpoint file. The current centralized checkpoint file creates a tight coupling between the Streams engine and every store, preventing stores (e.g., RocksDB with remote storage) from atomically persisting their data and the corresponding changelog offset together.

TLDR: Introduces a dead-letter queue (DLQ) mechanism for Kafka Streams, routing records that trigger processing exceptions to a configurable Kafka topic instead of failing the stream or silently dropping them. The existing exception handlers offer only two choices—fail the application or skip the record—neither of which is suitable for production workloads where faulty records should be isolated for inspection and reprocessing.

TLDR: KIP-1033 adds a `ProcessingExceptionHandler` interface to Kafka Streams (configured via `processing.exception.handler`) that intercepts uncaught exceptions thrown during `process()` calls in DSL or Processor API topologies, with built-in `LogAndFail` (default) and `LogAndContinue` implementations. Previously, unhandled processing exceptions terminated the `StreamThread`, requiring developers to wrap all processing logic in try-catch blocks—an error-prone approach that does not scale across large topologies.

TLDR: Upgrades Kafka's REST API layer (Connect, broker REST endpoints) from legacy javax.* (JavaEE) packages to jakarta.* (Jakarta EE 10), enabling compatibility with Jetty 12, Spring 6, and other modern frameworks that have dropped javax support. Kafka 4.0 is a major version bump and an appropriate milestone to make this breaking package namespace change; continuing to use javax blocked dependency upgrades that delivered security fixes.

TLDR: Adds a configuration flag `offset.syncs.enabled` to `MirrorSourceConnector` to let operators disable the writing of offset translation records to the `mm2-offset-syncs` internal topic during mirroring. This is useful when MirrorCheckpointConnector is not deployed and offset sync overhead is unnecessary.

TLDR: KIP-1030 revises the constraints and default values for multiple broker and topic configurations in the Kafka 4.0 and 5.0 releases, such as tightening the minimum value for `segment.ms` and updating defaults to reflect modern operational best practices. The 4.0 major release provides a compatibility break point to fix defaults and constraints that have been suboptimal but could not be changed in minor releases.

TLDR: Publishes a Docker Official Image (DOI) for Apache Kafka under Docker Hub's `kafka` namespace, complementing the ASF-sponsored `apache/kafka` image introduced by KIP-975. This gives users a Docker-community-maintained, standards-compliant image alongside the existing ASF one.

TLDR: Adds `MockFixedKeyProcessorContext` and `TestFixedKeyRecordFactory` to Kafka Streams test-utils to enable unit testing of `FixedKeyProcessor` topologies introduced in KIP-478/KIP-820. The existing `MockProcessorContext` only supports the general `Processor` API; `FixedKeyProcessor` uses a different context type (`FixedKeyProcessorContext`) which had no test-utils analog, making unit tests for fixed-key topologies impossible without an integration test setup.

TLDR: Adds a sasl.oauthbearer.header.urlencode client configuration to opt into URL-encoding of clientID and clientSecret when constructing the Basic Authorization header for OIDC token endpoint requests. RFC 6749 §2.3.1 mandates URL-encoding these credentials in the Authorization header, but the KIP-768 implementation omitted this encoding, risking authentication failures with strict OIDC providers.

TLDR: Enables follower replicas to start fetching from the tiered storage offset (the remote log start offset) rather than always from local log start, when rebuilding auxiliary state (leader epoch cache, producer snapshots) from the leader. Without this, followers on tiered-storage-enabled topics redundantly fetch local segments that overlap with remote storage, slowing restoration.

TLDR: Extends the kafka-storage format and kafka-features upgrade tools to accept feature-specific flags (--feature transaction.version=X, --feature group.version=X, etc.) and introduces Transaction Version and Group Version as named features with their own upgrade lifecycle. Feature initialization was limited to metadata version, making it impossible to set or upgrade other feature levels independently, blocking the phased rollout of EOS v2 (KIP-890) and new consumer group protocol (KIP-848).

TLDR: Adds an --isolation flag to kafka-get-offsets.sh and extends OffsetSpec with an LSOSpec so that the last stable offset (LSO) can be fetched instead of the high watermark. Without this, the tool always uses IsolationLevel.READ_UNCOMMITTED, making it impossible to observe the true consumer-visible offset boundary for transactional topics.

TLDR: Adds a public `isMeasurable()` method to the `KafkaMetric` interface so callers can check measurability without relying on exception handling or Java reflection. Existing code either caught `ClassCastException` from `metric.measurableValue()` or used reflection to inspect the internal value provider, both of which are fragile and expensive.

TLDR: Adds a dedicated remote.fetch.max.wait.ms broker configuration for the DelayedRemoteFetchPurgatory timeout used when fetching from tiered (remote) storage, independent of the consumer's fetch.max.wait.ms. The shared fetch.max.wait.ms default (500ms) was too short for remote storage fetches during degraded conditions, causing excessive purgatory expiration and retry storms against the remote storage layer.

TLDR: KIP-1017 adds a `/health` REST endpoint to Kafka Connect workers that returns HTTP 200 only when the worker has fully completed startup (internal topic creation, catch-up read, cluster join) and is ready to handle requests. Kubernetes readiness probes and load balancers have no reliable way to determine whether a Connect worker has finished its multi-step startup sequence before routing traffic to it.

TLDR: Makes the MirrorMaker2 heartbeats topic name configurable via a new deafult.replication.policy.heartbeats.topic.name MirrorConnectorConfig property, defaulting to the existing heartbeats value. The hard-coded heartbeats topic name in DefaultReplicationPolicy caused naming collisions when multiple MM2 instances or other systems used the same topic name in the same cluster.

TLDR: Adds a new `max.connections.ssl` broker and listener configuration to cap the number of active TLS connections independently of plaintext connections. An SSL connection consumes ~100KB of memory vs ~250 bytes for plaintext, so the existing unified `max.connections` limit does not prevent OOM from TLS-heavy workloads.

TLDR: KIP-1014 introduces an "unstable feature" framework for Apache Kafka where in-progress features are gated behind a feature flag and isolated from production traffic until fully complete and stabilized. Large features developed across many PRs produce a cluster in a partially implemented state where incompatible RPC or metadata record changes can break mixed-version clusters or expose unfinished functionality to users.

TLDR: Proposes releasing Kafka 3.8 and 3.9 as minor releases before the 4.0 major release to allow time to complete KRaft feature parity (e.g., JBOD support) that the community depends on before removing ZooKeeper entirely. Without these intermediary releases, operators would be forced to upgrade to 4.0 before critical features are ready.

TLDR: Switches `kafka-configs.sh` to use `IncrementalAlterConfigs` instead of the deprecated `AlterConfigs` RPC when modifying broker configs, ensuring per-key updates rather than full config replacement. The old `AlterConfigs` semantics require sending all configs atomically, risking accidental resets of configs not included in the request.

TLDR: Introduces per-topic-partition bandwidth quotas as a new quota entity type, complementing the existing per-user/per-client quotas from KIP-13. When multiple partitions of a high-throughput topic land on the same broker, per-client quotas cannot cap the aggregate write rate to that topic across clients, allowing a topic to saturate broker I/O regardless of how the quota is configured.

TLDR: Adds broker-level throttle configurations (`replication.quota.throttled.rate`) that apply globally across all partitions undergoing replication catch-up or reassignment, without requiring per-topic quota configuration. Without global replication throttles, large-scale rebalances or broker recoveries saturate inter-broker bandwidth and spike producer latency.

TLDR: Proposes a `ParKa` format that serializes Kafka record batches in Apache Parquet columnar format rather than row-by-row Avro/JSON/Protobuf. Columnar encoding yields dramatically better compression ratios for large batches and enables predicate pushdown for consumers reading from tiered storage.

TLDR: Removes Kafka's compile-time and runtime dependencies on the Java `SecurityManager` API (including `AccessController`) from clients, core, and connect-runtime modules. JEP-411 deprecated `SecurityManager` in Java 17 and Java 23 already throws `UnsupportedOperationException` for legacy APIs when no `SecurityManager` is configured, making these calls a forward-compatibility hazard.

TLDR: Exposes the `EARLIEST_LOCAL_OFFSET` (`-4`) and `TIERED_OFFSET` ListOffsets special integers in the `kafka-get-offsets` CLI tool, and makes `TieredOffset` an accessible public constant via `UnifiedLog`. KIP-405's tiered storage introduced a local log start offset exposed by the `ListOffsets` API, but the `kafka-get-offsets` tool had no way to request it, and the offset constant was not publicly accessible.

TLDR: Enforces the `tasks.max` connector configuration so that a connector can never run more tasks than the configured upper bound, even if its `taskConfigs()` method returns more entries than allowed. Kafka Connect previously passed all generated task configs to the connector regardless of `tasks.max`, allowing connectors to silently exceed their resource quota.

TLDR: Extends RemoteStorageManager#fetchLogSegment with an optional nextRemoteLogSegmentMetadata parameter, allowing RSM implementations to begin pre-fetching the next segment before the current one is fully consumed. Without segment boundary signaling, RSM pre-fetching was limited to within the current segment, causing latency spikes at segment transitions during sequential remote reads.

TLDR: Adds a `RemoteStorageManager.fetchLogSegmentData(metadata, indexes)` batch API to retrieve all segment indexes (offset, time, transaction, leader epoch) in a single remote storage call alongside the log data. Currently each index type is fetched with a separate `fetchIndex()` call, multiplying remote storage round-trips per consumer fetch.

TLDR: Adds a `CurrentControllerId` metric exposed on every broker and controller node that reports the node ID of the active KRaft (or ZooKeeper) controller, or -1 if unknown. Identifying the current active controller requires either checking each node's `ActiveControllerCount` metric individually or parsing metadata logs; there is no single metric queryable from any node that identifies which node is the controller.

TLDR: Adds a `ListClientMetricsResources` admin API and corresponding `kafka-configs.sh` support to enumerate all client metrics subscription resources created by KIP-714. Client metrics resources are named configurations unassociated with any other Kafka entity, so without this there is no way to discover what subscriptions exist.

TLDR: Proposes exposing server-side consumer lag metrics (committed offset vs. log end offset per group-partition) directly from brokers via the metrics framework. This eliminates the need for external lag monitoring tools that must independently query both consumer group offsets and partition end offsets.

TLDR: Extends `WindowRangeQuery` in the Kafka Streams Interactive Queries API to support dual-key range queries (`keyFrom`, `keyTo`, `timeFrom`, `timeTo`), unifying it with `WindowKeyQuery`. The existing `WindowRangeQuery.fetchAll(timeFrom, timeTo)` fetches all keys in a time window but cannot restrict the key range, forcing clients to post-filter.

TLDR: Implements Pre-Vote in KRaft: a partitioned controller first canvasses the cluster to check whether it would win a majority vote before incrementing its epoch and sending a real VoteRequest. Without Pre-Vote, a follower partitioned from the leader continuously increments its epoch and sends disruptive VoteRequests upon reconnection, forcing unnecessary leader elections and causing flip-flopping leadership in certain network partition scenarios.

TLDR: Extends the POST /connectors REST endpoint with an optional initial_offsets field, allowing connector creation and offset initialization in a single atomic request. Previously, setting initial offsets required a three-step sequence: create in STOPPED state, PATCH offsets, then set RUNNING.

TLDR: Enhances `ListTransactions` and `DescribeTransactions` APIs with new filter options (filter by producer ID, duration threshold) and adds a `lastTransactionStartTimeMs` field to transaction metadata. The existing tooling introduced by KIP-664 lacked the ability to narrow responses by producer or flag suspiciously long-running transactions.

TLDR: Adds an allowed.paths configuration to FileConfigProvider and DirectoryConfigProvider, restricting which filesystem paths those providers may read. In security-sensitive Connect deployments, unrestricted file access allows any connector configuration to read arbitrary files on the worker host.

TLDR: Introduces TimestampedKeyQuery and TimestampedRangeQuery as explicit IQv2 query types for timestamped key-value stores, and fixes KeyQuery/RangeQuery to always return plain V from both kv-store and ts-kv-store. Previously, KeyQuery returned ValueAndTimestamp<V> against a ts-kv-store, creating type-safety issues and unintuitive API behavior.

TLDR: Adds num-open-iterators and thread-open-iterators metrics per StateStore in Kafka Streams to expose the count of currently open (potentially leaked) store iterators. Iterator leaks on RocksDB stores pin in-memory blocks and cause memory growth, but the existing block-cache-pinned-usage metric reports shared block cache usage without identifying which store is responsible.

TLDR: Adds a `StandbyUpdateListener` interface to Kafka Streams that fires callbacks when standby tasks begin, complete, or make progress restoring state from changelog topics. Currently, `StateRestoreListener` only covers active task restoration, leaving standby task state opaque to operators.

TLDR: Proposes adding static (manual) task assignment to Kafka Connect's distributed mode, allowing operators to pin connectors/tasks to specific worker nodes instead of relying on the existing rebalancing algorithms. The current eager and cooperative scheduling algorithms rebalance all tasks on any cluster membership change, causing unnecessary task restarts when operators only need to change a single task's placement.

TLDR: Proposes building cross-cluster replication natively inside the Kafka broker (as opposed to external MirrorMaker processes), using new internal replication protocol and management APIs. MirrorMaker 1 and 2 run as separate processes that can independently fail, require separate deployment, and cannot leverage broker-internal optimizations (zero-copy transfers, direct partition assignment knowledge).

TLDR: Adds withDescendingKeys() and withAscendingKeys() methods to RangeQuery and introduces the ResultOrder enum, enabling reverse-ordered range and all-keys queries against RocksDB and InMemoryKeyValueStore (both of which maintain sorted key order). IQv2 previously had no mechanism to retrieve results in descending key order.

TLDR: Adds a pluggable compression interface to Kafka by reserving 4 additional bits in the record batch attributes for a compression plugin ID, allowing third-party compression codecs to be loaded at runtime without modifying or rebuilding Kafka. Currently adding a new compression algorithm requires forking Kafka source and recompiling the distribution.

TLDR: Proposes a new Java consumer API mode where partitions not being revoked during a cooperative rebalance continue processing at full speed while the application asynchronously drains revoked partitions before committing. The current asynchronous processing pattern blocks the entire poll loop during rebalance to prevent offset commit races, stalling throughput on all partitions even those unaffected by the rebalance.

TLDR: Injects `SslPrincipalMapper` and `KerberosShortNamer` into custom `KafkaPrincipalBuilder` implementations via a new `configure()` callback with broker security configs. Currently these mappers are null inside custom principal builders, preventing them from applying the SSL principal mapping rules configured on the broker.

TLDR: Extends Kafka Connect's topic management to accept a custom Admin implementation (via ForwardingAdmin), routing topic creation through organization-managed resource management systems instead of calling AdminClient directly. Organizations that enforce centralized capacity controls cannot allow Connect to bypass those systems by creating topics with direct AdminClient calls.

TLDR: Adds a `stopped` initial state option to the Kafka Connect `POST /connectors` REST endpoint so newly created connectors start in the `STOPPED` state rather than immediately running. This enables zero-downtime connector migration between Connect clusters by creating the connector on the target cluster before starting it.

TLDR: Adds separate PID files and stop scripts for KRaft controller and broker processes so they can be independently stopped when co-located on the same machine. Previously a single `kafka-server-stop.sh` script signaled all Kafka processes indiscriminately, making it impossible to stop just the controller without killing the co-located broker.

TLDR: Adds per-partition metrics for `MessagesInPerSec`, `BytesInPerSec`, and `BytesOutPerSec` alongside existing topic-level metrics. Topic-level throughput metrics cannot reveal partition hot-spots, making it impossible to detect key skew or diagnose which specific partition is causing ISR shrinkage or broker overload.

TLDR: Adds a scope=cluster query parameter to PUT /admin/loggers/{logger} in Kafka Connect's REST API, broadcasting log-level changes to every worker in the cluster in a single request. The existing per-worker API required one request per worker, making cluster-wide log-level changes during debugging cumbersome and impossible through a load balancer.

TLDR: Publishes an official Apache Kafka Docker image to Docker Hub under the ASF Docker profile, targeting JVM-based Kafka. Apache Kafka had no official Docker image, leaving users to rely on third-party images with inconsistent configurations and no ASF endorsement.

TLDR: Introduces an experimental Docker image that packages a GraalVM ahead-of-time compiled Kafka broker binary, achieving sub-second startup (~110–140 ms) and ~250–540 MB RAM usage versus ~1150 ms and ~1 GB for the JVM image. Long JVM startup times and memory overhead slow down developer iteration cycles involving ephemeral broker instances in unit tests.

TLDR: Corrects ReplicationBytesInPerSec and ReplicationBytesOutPerSec MBeans to report per-topic metrics (adding a topic tag) rather than only aggregate broker-level metrics, aligning implementation with documentation. Tools like Cruise Control assumed these metrics were per-topic but the implementation only tracked broker-level aggregates, producing incorrect workload models.

TLDR: Adds version and commitId as tags on the existing kafka.server:type=app-info metric's start-time-ms gauge, so Prometheus and similar numeric-metric systems can extract the running broker version via label queries. The existing app-info metric exposed version as a string attribute, which Prometheus cannot scrape as a numeric time series.

TLDR: Adds replication-record-lag metrics (max/min/avg) to MirrorMaker2's MirrorSourceTask, computed as the difference between the source partition's LEO and the last replicated offset (LRO). No built-in metric existed to directly quantify how many records remained to be replicated, forcing operators to infer lag from byte-lag proxies.

TLDR: Deprecates GET /connectors/{connector}/tasks-config (added by KIP-661) in Kafka 3.7 and removes it in 4.0, as GET /connectors/{connector}/tasks already returns task configurations. Two endpoints exposing functionally identical data creates confusion and unnecessary API surface.

TLDR: Introduces MultiVersionedRangeQuery for IQv2, supporting key-range queries (bounded and unbounded) against versioned KeyValueStores with optional timestamp bounds. The original IQv2 range query types only targeted the latest value and could not retrieve historical versions within a key range from versioned stores.

TLDR: KIP-968 adds a `MultiVersionedKeyQuery<K, V>` class and `VersionedRecordIterator<V>` interface to Kafka Streams IQv2, enabling single-key queries against versioned state stores that return all versions within a time range (with optional upper/lower bounds and result ordering). KIP-960 added single-timestamp point lookups; this KIP extends IQv2 to support range-over-time queries including tombstone representation via a `validTo` field on `VersionedRecord`.

TLDR: Introduces Eligible Leader Replicas (ELR), a per-partition set of replicas that lost data in an unclean shutdown but are still eligible for leader election under KRaft, providing minISR-1 tolerance to data-loss unclean shutdowns. The last-replica-standing scenario allowed a single replica that suffered an unclean shutdown and lost committed data to be re-elected as leader, causing surviving replicas to truncate their logs and produce cluster-wide committed data loss.

TLDR: Adds sync.full.acl.enabled to MirrorMaker2 configuration, enabling full replication of topic read/write ACLs, consumer group ACLs, and user SCRAM credentials to the target cluster for disaster recovery standby scenarios. By default MM2 downgrades ALLOW ALL ACLs to ALLOW READ to prevent writes to the mirror, which breaks transparent failover when the target cluster needs to become the active cluster.

TLDR: Adds tiered storage operational metrics covering upload rate, deletion rate, segment copy latency, auxiliary state build time, and remote fetch latency to the existing `RemoteStorageMetrics`. Without these, operators cannot tell whether the remote log manager is keeping pace with log growth or falling behind.

TLDR: Changes Kafka Streams left-join and outer-join operators (KStream-KStream, KStream-KTable, KStream-GlobalTable, KTable-KTable foreign-key) to no longer drop records with null keys, instead passing null to ValueJoiner for the absent side. The existing strict null-key rejection was inconsistent with the defined left/outer join semantics of retaining the record when no match is found.

TLDR: KIP-960 adds a `VersionedKeyQuery<K, V>` class to Kafka Streams IQv2 supporting single-key lookups against versioned state stores (`VersionedKeyValueStore`) with an optional `asOf(Instant)` timestamp to retrieve the value valid at a specific point in time. Versioned state stores (introduced in KIP-889) expose time-varying key-value state but lacked an interactive query interface, making the stored temporal history inaccessible from outside the topology.

TLDR: Adds a `BooleanConverter` class to the `org.apache.kafka.connect.converter` package that serializes/deserializes `BOOLEAN` schema types for Kafka Connect. The Boolean SerDe added in KIP-907 was only available for raw Kafka clients, leaving Connect without a native boolean converter.

TLDR: Introduces configurable bandwidth quotas for the Remote Log Manager (RLM) upload and fetch paths in Tiered Storage to bound the CPU and network impact of segment uploads. When Tiered Storage is enabled on existing large topics, the RLM schedules concurrent upload tasks for all eligible segments simultaneously, consuming significant CPU and degrading producer latencies.

TLDR: Extends the default.dsl.store StreamsConfig to cover custom state store types and stream-stream joins (both windowed and the internal KeyValueStore used for certain join types), which were previously hardcoded to RocksDB or in-memory stores. Without this, there was no way to use a fully custom store implementation across all DSL operators, especially the additional KeyValueStore in certain stream-stream join topologies.

TLDR: KIP-952 changes the Remote Log Manager to retroactively generate missing producer snapshot files when uploading a log segment to remote storage if the segment predates Kafka 2.8.0 (which introduced segment-aligned snapshot files). Users upgrading from pre-2.8.0 Kafka and enabling Tiered Storage immediately encounter `NullPointerException`s because old segments lack snapshot files that the RSM API contract requires, blocking all remote log uploads.

TLDR: Embeds new leader information (leader ID, epoch, host/port) directly in ProduceResponse and FetchResponse error payloads when a broker knows the current leader, eliminating or reducing the need for a separate Metadata RPC round-trip after a leadership change. The existing leader discovery path required an asynchronous Metadata refresh plus retry backoff before the client could resume producing or fetching, significantly increasing end-to-end latency during leader elections and cluster rolls.

TLDR: Adds the ability to disable Tiered Storage on a per-topic basis after it has been enabled, including draining remote segments back to local storage before disabling. KIP-405 introduced Tiered Storage enablement but provided no mechanism to reverse it, leaving operators without a path to recover topics from tiered storage or roll back the feature.

TLDR: Adds replication.policy.internal.topic.separator.enabled (default true) to DefaultReplicationPolicy, allowing users to opt out of using replication.policy.separator for naming internal offset sync and checkpoint topics. KIP-690 introduced the separator into internal topic naming, breaking backward compatibility for users who upgraded and had existing internal topics named with the old convention.

TLDR: Introduces an internal.topics.prefix StreamsConfig that prepends a custom string to all Kafka Streams internal topic names independently of application.id. Multi-tenant clusters often enforce a topic namespace prefix tied to team ACLs, but application.id serves double duty as both consumer group ID and topic prefix, preventing them from being set independently.

TLDR: Proposes standardizing and documenting the exceptions thrown by `KafkaConsumer` public APIs to give callers clear, actionable error types. Currently consumer APIs throw a mix of undocumented runtime exceptions making it difficult for application code to handle errors correctly.

TLDR: Refactors the `KafkaConsumer` threading model by moving heartbeating and coordinator communication to a dedicated background thread while keeping application callbacks on the calling thread. The existing implementation mixes application-thread and background-thread execution in ways that make the code fragile, hard to reason about, and prone to subtle concurrency bugs.

TLDR: Adds CI/CD pipeline support and testing for Apache Kafka on IBM Power (ppc64le) architecture. Kafka had no official support or validated builds for ppc64le, which is widely used in banking and HPC industries running on IBM infrastructure.

TLDR: Changes RangeQuery.withRange(K lower, K upper) to treat null bounds as absent (using Optional.ofNullable), so null lower/upper bounds implicitly widen to unbounded rather than throwing or using null as a literal key. Callers receiving nullable query parameters from HTTP clients had to write explicit null-checking logic before constructing a RangeQuery.

TLDR: Proposes a broker-side RecordValidationPolicy extension point that intercepts records before append and can reject them with InvalidRecordException. There is currently no broker-enforced mechanism to prevent misconfigured clients from bypassing schema registry serializers and producing incorrectly formatted records.

TLDR: Enables Kafka brokers to act as a participant (not just coordinator) in an external two-phase commit (2PC) protocol by adding a `PrepareTransaction` API phase before the existing `EndTransaction`. Without native 2PC participation support, applications needing atomic cross-system transactions (Kafka + external DB) must implement fragile custom recovery logic outside the broker.

TLDR: Adds a set of KRaft-specific metrics including TimedOutBrokerHeartbeatCount, EventQueueOperationsStartedCount, NewActiveControllersCount, CurrentMetadataVersion, HandleLoadSnapshotCount, and LatestSnapshotGeneratedBytes/AgeMs. KRaft mode lacked sufficient observability into controller health, quorum performance, and metadata snapshot lifecycle.

TLDR: Adds a new log.message.timestamp.before.max.ms config (and optionally log.message.timestamp.after.max.ms) for brokers/topics to reject producer records whose CreateTime timestamp is too far in the past or future, returning INVALID_TIMESTAMP. The existing log.message.timestamp.difference.max.ms config was a symmetric bound that could not distinguish past vs. future violations, and its default of Long.MAX_VALUE allowed producers misconfigured with nanosecond timestamps to corrupt log retention behavior.

TLDR: KIP-936 proposes a configurable cap on the maximum number of active producer IDs (PIDs) tracked by `ProducerStateManager` per broker, with throttling or rejection when the limit is reached. Since KIP-679 made idempotent producers the default, every producer instance is assigned a PID; a large number of short-lived or restarting producers can cause `ProducerStateManager` to accumulate unbounded in-memory state, risking OOM errors.

TLDR: Extends the `AlterConfigPolicy` interface to receive both the proposed config changes and the current (existing) configuration values, so policies can validate incremental updates in context. After KIP-339 introduced `IncrementalAlterConfigs`, policy implementations only see the delta, not the full resulting configuration, making context-aware validation impossible.

TLDR: Introduces a `DeleteTopicPolicy` plugin interface (analogous to `CreateTopicPolicy` from KIP-108) that brokers invoke before executing a topic deletion to allow operators to veto or control the deletion. Without a deletion policy, operators have no server-side safeguard to prevent accidental deletion of critical internal topics (e.g., `__consumer_offsets`, Connect internal topics) even when coarse-grained ACLs grant delete permission.

TLDR: Adds source-record-poll-error-total and source-record-poll-error-rate metrics at task granularity in kafka.connect:type=task-error-metrics to track failures in SourceTask#poll(). No existing metric distinguished between failures during polling from the source system versus failures when producing to Kafka, forcing operators to rely on log inspection.

TLDR: Introduces Share Groups to Kafka, providing queue semantics with cooperative multi-consumer consumption, per-message acknowledgment, delivery counts, and redelivery of unacknowledged records. Classic consumer groups couple parallelism to partition count and lack per-message acknowledgment, making them unsuitable for work-queue patterns where tasks are independent, consumers vary dynamically, and failed work must be redelivered.

TLDR: Proposes a batch-header attribute bit to optionally omit the per-message attributes field (currently always present but unused since message format v1) from records, reducing per-record overhead. Discarded because space savings do not outweigh message conversion costs and planned v3 format changes will address this more comprehensively.

TLDR: Renames tiered storage metrics `RemoteBytesIn`/`RemoteBytesOut` to `RemoteBytesRead`/`RemoteBytesWritten` to align with Kafka's standard `BytesIn`/`BytesOut` semantics (in=written-to, out=read-from). The original naming was inverted relative to broker convention, causing operator confusion in dashboards.

TLDR: Proposes making Kafka (KRaft) resilient to log directories becoming full by detecting the condition, fencing the affected partitions, and alerting operators rather than crashing. Full log directories currently cause uncontrolled broker behavior; controlled fencing prevents data loss and enables operators to take remediation actions.

TLDR: Adds a `--human-readable` flag to the `kafka-metadata-quorum` CLI tool that converts `LastFetchTimestamp` and `LastCaughtUpTimestamp` fields from Unix epoch milliseconds to human-readable relative delay strings (e.g., `366 ms ago`). Raw epoch timestamps in the quorum replication status output require manual conversion to assess voter lag.

TLDR: Proposes a new `acks=min.insync.replicas` producer configuration value that dynamically binds the acks requirement to the broker's current `min.insync.replicas` setting for the target topic. This eliminates the need to keep producer `acks` configuration in sync with per-topic `min.insync.replicas` values as they change.

TLDR: Adds rack-aware task assignment to Kafka Streams' HighAvailabilityTaskAssignor by preferring to assign StreamTasks to Streams clients located in the same rack as the TopicPartition replicas they need to read. Without rack awareness, Streams tasks could be assigned to instances in a different availability zone than the partition leader, incurring cross-rack network costs and higher fetch latency.

TLDR: KIP-924 promotes the `task.assignor.class` configuration in Kafka Streams to a public API, allowing users to select between `HighAvailabilityTaskAssignor` and `StickyTaskAssignor` (or supply a custom implementation). KIP-441 introduced `HighAvailabilityTaskAssignor` as the new default but the existing `StickyTaskAssignor` remains preferable in some use cases, and the existing backdoor config was internal with no stability guarantee.

TLDR: Adds a gracePeriod parameter to Joined to buffer stream-side records in a stream-table join, allowing them to match the correct versioned table entry rather than the latest value at arrival time. Without buffering, stream records that arrive slightly before a table update join with the previous version, producing incorrect results even when a versioned table is used.

TLDR: Adds per-partition metrics for `MessagesInPerSec`, `BytesInPerSec`, and `BytesOutPerSec` at the broker level. Topic-level aggregates mask partition-level traffic skew, making hot-spot detection and ISR instability diagnosis require external tooling that cross-references partition leaders with topic metrics.

TLDR: KIP-921 adds support for OpenJDK CRaC (Coordinated Restore at Checkpoint), allowing Kafka brokers and clients to checkpoint their JVM state to disk and restore it with near-zero startup time. Kafka's long JVM warmup time (connection establishment, metadata loading, JIT compilation) makes it a poor fit for elastic scaling scenarios where fast restart is critical.

TLDR: KIP-919 allows `AdminClient` to communicate directly with KRaft controller quorum nodes (not just broker nodes) when a new `bootstrap.controllers` config is specified, enabling admin operations like `DescribeQuorum` and controller-targeted `IncrementalAlterConfigs` without involving brokers. It also introduces a `ControllerRegistration` RPC so KRaft controllers register their endpoints and metadata (including ZK migration readiness) with the active controller, making controller topology discoverable via the admin API.

TLDR: Adds a `TopicAndGroupListener` API to MM2 that notifies registered listeners when the set of replicated topics or consumer groups changes. Currently there is no way to observe which topics and groups MM2 is actively replicating without parsing remote topic names or using monitoring heuristics that break with `IdentityReplicationPolicy`.

TLDR: Allows RemoteStorageManager.copyLogSegmentData() to return an optional opaque byte[] of custom metadata that is stored in RemoteLogSegmentMetadata and returned on every fetch. RSM implementations need to persist implementation-specific data (e.g., bucket name for load-balanced S3 placement) alongside standard segment metadata, but the existing API had no extension point for this.

TLDR: Adds a flow.context MDC key (e.g., [primary→backup]) and embeds the replication flow in Connect thread names when running MirrorMaker2 in dedicated mode. Multiple MM2 replication flows share identical connector and task names, making log lines and thread names indistinguishable across flows during troubleshooting.

TLDR: Introduces versioned record types and downgrade support for the transaction coordinator (`__transaction_state`) and group coordinator (`__consumer_offsets`) by adding version headers and unknown-record skipping logic. Without this foundation, adding any new record field or type to these internal topics permanently prevents downgrading to an older broker version.

TLDR: Enables versioned key-value stores (KIP-889) to be used as backing stores for `KTable` in DSL stream-table joins, so out-of-order stream records are joined against the table value at the record's event timestamp rather than the latest value. Without versioned stores, stream-table joins always use the latest table value regardless of event time, producing incorrect results for late-arriving stream records.

TLDR: KIP-913 proposes adding a `KafkaProducer.getCluster(String topic, long maxBlockTimeMs)` public method that triggers an immediate metadata fetch and returns the resulting `Cluster` object, allowing applications to warm up the metadata cache before handling their first request. On cold start, the first `KafkaProducer.send()` incurs the full metadata-fetch latency because the local cache is empty; pre-fetching metadata during application initialization moves this cost out of the critical path.

TLDR: Adds a `source` tag to `MirrorSourceConnector` metrics (alongside the existing `target`, `topic`, `partition` tags) to identify the source cluster alias. With `IdentityReplicationPolicy`, topic names do not contain the source alias, making source attribution impossible in multi-cluster mirroring setups.

TLDR: Adds a SourceTask#updateOffsets() method allowing source connectors to commit updated offsets to the Connect offset store without producing any records to Kafka. Without this, offsets only advance when records are produced, causing offset staleness during quiet periods in the source system and risking inability to resume from the correct position if the source purges its change log.

TLDR: Changes Kafka client construction to tolerate transient DNS resolution failures by deferring the fatal ConfigException and retrying resolution in the background during the connection establishment phase. A client whose bootstrap.servers hostname cannot be resolved at construction time immediately throws a ConfigException and dies, even if the DNS entry is about to become available—an especially severe problem in containerized environments with slow DNS propagation.

TLDR: Adds `BooleanSerializer`, `BooleanDeserializer`, and `Serdes.Boolean()` to the public `org.apache.kafka.common.serialization` package. Boolean was the only primitive type missing from Kafka's built-in serialization library, requiring users to implement their own or convert to byte/integer.

TLDR: Establishes migration guidelines and compatibility rules for moving Kafka CLI tools from the `kafka.tools`/`kafka.admin` packages in the `core` module to `org.apache.kafka.tools` in the dedicated `tools` module. The ongoing module-split initiative (KAFKA-14524/14525) risks breaking users who invoke tools by their fully-qualified class name or via SPI arguments, so explicit compatibility contracts are needed.

TLDR: Proposes a `BrokerInterceptor` plugin point on the broker side that allows platform teams to inject cross-cutting logic (schema validation, serialization enforcement, privacy checks) into the produce/consume path without requiring changes to polyglot client libraries. Enforcing platform standards across diverse client languages today requires maintaining duplicate library extensions in every supported language, which is operationally expensive and error-prone.

TLDR: KIP-904 changes `KTable.groupBy()` to emit a single combined event when the grouping key is unchanged (carrying both the old and new value together) instead of two separate tombstone/insert events, guaranteeing the subtractor is applied before the adder in the downstream aggregate. With the two-event approach, a stream-table join could observe the intermediate state after the subtractor fires but before the adder fires, producing a transiently inconsistent aggregate value that is visible to downstream consumers.

TLDR: Enforces in KRaft mode that an AlterPartition request is rejected (INELIGIBLE_REPLICA) if any proposed ISR member has a broker epoch older than the epoch tracked by the leader at the time of the Fetch request. A race where a broker reboots with a new empty disk, gets re-registered, and is then added to the ISR by a stale AlterPartition from before the fence was lifted could allow that broker to become leader with an empty log, causing data loss.

TLDR: Upgrades Kafka's ZooKeeper dependency from 3.6.3 (EOL December 2022) to 3.8.2. ZooKeeper 3.6.3 no longer receives security or bug-fix patches; continuing to ship it exposes Kafka deployments to unpatched vulnerabilities and prevents adoption of improvements in the 3.8.x release line.

TLDR: Extends kafka-storage.sh format with an --add-scram option that writes UserScramCredentialsRecord entries into the bootstrap.checkpoint, enabling SCRAM inter-broker authentication from the first cluster startup in KRaft mode. In ZooKeeper mode, SCRAM credentials could be seeded into ZooKeeper before brokers started; no equivalent mechanism existed for KRaft's __cluster_metadata bootstrap.

TLDR: Allows Kafka clients to fall back to the bootstrap servers list and re-fetch cluster metadata when all currently known brokers become unreachable, rather than failing permanently. This solves the case where the entire broker set is replaced (e.g., IP changes, full cluster migration) and the cached metadata is entirely stale.

TLDR: Replaces reflective class scanning during Connect worker startup with a manifest-based plugin discovery mechanism where plugin JARs include a pre-generated index of their classes. Reflective scanning loads every class on the classpath to find connectors, significantly delaying worker startup and provisioning in cloud environments.

TLDR: Sets a new minimum baseline for Kafka wire protocol API versions at Apache Kafka 2.1 (released 2018), removing support for all older protocol versions in Kafka 4.0. Maintaining backward compatibility with 9+ years of protocol API versions added significant code complexity and testing matrix size with diminishing value as modern clients stopped using them.

TLDR: Proposes dynamically reloading the partition count of `__consumer_offsets` at runtime after it is increased, without requiring a rolling broker restart. Currently the new partition count is only picked up on broker startup, so expanding the consumer offsets topic requires a disruptive rolling restart to redistribute group coordinators.

TLDR: Migrates MirrorMaker2 topic configuration sync from the deprecated AlterConfigs RPC to IncrementalAlterConfigs, controlled by a use.incremental.alter.configs setting (default requested). The AlterConfigs API replaces the entire topic config, causing MirrorMaker2 to clear configurations it was not supposed to replicate (e.g., throttling replicas set by Cruise Control on the target cluster).

TLDR: Adds nullable struct support to the Kafka RPC protocol schema, enabling struct fields to be null on the wire (serialized as INT8 -1/VARINT 0 presence byte before the struct bytes). This is needed for KIP-848's ConsumerGroupHeartbeat response, which must be able to return null assignment to indicate no change, rather than re-sending the full assignment every heartbeat.

TLDR: Introduces transactional semantics for RocksDB-backed Kafka Streams state stores, ensuring state is only persisted to disk once the corresponding changelog offset is durably committed, bounding state restore under EOS to under 1 second. Under the current design, state is written to the local store before changelog commits complete, so EOS crash recovery must replay the entire changelog from scratch, causing restore times proportional to changelog size that in production can span multiple days.

TLDR: Allows Kafka Connect workers to load and run multiple versions of the same connector plugin simultaneously, decoupling connector version installation from migration of individual connector instances. Without multi-version support, upgrading a connector across all tasks requires a simultaneous cutover with no incremental rollback path, and running two connector versions requires two separate Connect clusters.

TLDR: Adds server-side transaction defense in the broker to reject out-of-order or delayed transactional writes that could cause hanging transactions (stalled LSO), incorrect transaction completion, or EOS violations. Network partitions and client bugs can deliver delayed `AddPartitionsToTxn` or `EndTxn` requests after epoch boundaries, leading to partially-committed state that blocks read-committed consumers and prevents log compaction.

TLDR: KIP-889 introduces versioned (multi-version) state stores in Kafka Streams that retain multiple timestamped values per key, enabling the store to answer point-in-time lookups. Existing state stores keep only the latest value per key, making stream-table joins produce incorrect results for out-of-order records because the correct historical table value is no longer available.

TLDR: Extends DescribeAclsRequest (v4) and DescribeClientQuotasRequest (v2) to accept multiple filter sets in a single request, returning batched results. Management tooling that operates on many users/groups had to issue one RPC per entity, incurring unnecessary per-request overhead and serializing what could be a parallel server-side operation.

TLDR: Adds EnvVarConfigProvider, a new ConfigProvider implementation that reads values from environment variables with an optional allowlist.pattern regex filter. No built-in mechanism existed to inject secrets or environment-specific values from process environment variables into Kafka client or Connect configurations.

TLDR: Introduces a fluent builder API for `KafkaProducer` and `KafkaConsumer` with named methods, typed enum parameters, and Javadoc for each configuration option. The existing constructor takes an untyped `Properties` map, offering no IDE autocomplete, no type safety, and no discoverability of valid configuration values.

TLDR: Proposes a new `DescribeBroker` or extended `ApiVersions` response that explicitly includes the broker's Kafka version string and name. Currently clients infer broker version by correlating `ApiVersions` response ranges with known per-version API changes, which breaks when a release adds no new API versions.

TLDR: Adds default.client.supplier to StreamsConfig, allowing a custom KafkaClientSupplier class to be specified via configuration rather than requiring code changes to the KafkaStreams constructor call. Existing applications that want to swap the client supplier (e.g., for tracing or testing) must modify every instantiation site; a config-driven approach enables the change without touching application code.

TLDR: Modifies the Connect REST API's connector create/update endpoints to abort (rather than proceed) when configuration validation exceeds the request timeout, and proposes a per-request timeout override. The existing behavior would create/update the connector after validation completed even if the REST response had already timed out, resulting in phantom connectors that users did not know existed.

TLDR: Extends the Kafka consumer partition assignment algorithms to prefer assigning partitions to consumers whose rack matches the replica rack, minimizing cross-AZ fetches when the replication factor is lower than the number of AZs. The existing rack-aware fetch (KIP-392) only helped when every AZ had a local replica; with fewer replicas than AZs some consumers were always assigned partitions with no local replica.

TLDR: KIP-880 proposes a `KafkaPrincipalBuilder` implementation that extracts SPIFFE URIs from the X.509 SAN (Subject Alternative Name) extension of mTLS client certificates and returns them as `KafkaPrincipal` objects for use in ACL rules. Istio-managed microservices in Kubernetes use SPIFFE-based SVID certificates for workload identity, but Kafka's existing mTLS principal extraction only reads the `CN` field, forcing operators to configure separate authentication mechanisms rather than reusing the Istio-provided identity.

TLDR: Extends Kafka's rack awareness to support hierarchical multi-level rack topologies (e.g., host → rack → zone → datacenter), so partition replicas are distributed across the most fault-isolated units possible. The current single-level `broker.rack` model cannot express datacenter or zone boundaries for stretch cluster deployments.

TLDR: Enables Kafka Streams to detect when user input topic partition counts have changed and automatically resize internal repartition and changelog topics to match. Streams currently treats a partition count change as a fatal topology mismatch and shuts down, preventing any horizontal scaling of partitioned topologies without a full state migration.

TLDR: Adds a MetricsRegistry to the plugin configuration context so that Kafka plugins (partitioners, interceptors, Connect connectors, etc.) can register metrics that inherit parent component tags and reporters. Plugins that created their own Metrics instances got no parent tags, no reporter integration, and caused failures with singleton reporters like CruiseControlMetricsReporter when instantiated multiple times.

TLDR: Adds time-based (wall-clock interval) triggering for KRaft metadata snapshot generation, complementing the existing byte-threshold trigger. Without a time-based trigger, clusters with low metadata change rates may go very long periods without snapshots, causing slow controller startup due to full metadata log replay on boot.

TLDR: Adds GET/PATCH/DELETE /connectors/{connector}/offsets REST endpoints to Kafka Connect, enabling administrators to read, modify, and reset connector offsets without direct access to the internal offset topic. No official API existed for offset inspection or manipulation, forcing operators to write custom tooling against the internal __connect-offsets topic.

TLDR: Proposes `ExceptionHandlingDeserializer`, `RetryDeserializer`, and `PipeDeserializer`/`PipeSerializer` wrappers that compose deserialization with retry logic, dead-letter routing, and chained format conversion. Consumer applications today have no standard mechanism to handle deserialization exceptions without catching raw exceptions and implementing retry/skip logic manually.

TLDR: Proposes adding retention.max.eventtime.ms, a new retention policy that truncates log segments once the maximum event timestamp in the segment exceeds the configured age, independent of broker wall-clock time. Existing time-based retention compares append time to wall clock, which aggressively deletes data during reprocessing scenarios when Kafka Streams re-ingests old events.

TLDR: Enriches Kafka Streams state restoration with additional metrics (bytes/records restored, restoration lag) and new callbacks in `StateRestoreListener` covering batch-level progress for active tasks. Previously, visibility into restoration progress was minimal, making it hard to estimate restoration completion time or trigger operational actions.

TLDR: Introduces metadata transactions in KRaft so the controller can atomically commit a set of metadata records larger than the Raft fetch size limit (currently hard-coded at 8 KB) by spanning multiple batches with transaction start/end markers. Creating a topic with thousands of partitions generates a `TopicRecord` + N `PartitionRecord`s that can exceed 8 KB, but partial commits during a controller failover would leave the metadata store in an inconsistent state.

TLDR: Defines the migration path and new broker/controller APIs to move a Kafka cluster from ZooKeeper-based metadata to KRaft quorum without partition availability impact, writing metadata to both stores simultaneously during the bridge period to allow rollback. Completing KIP-500 required a concrete, reversible migration procedure since running ZooKeeper in production clusters has significant operational overhead and single-point-of-failure concerns.

TLDR: Adds --bootstrap-server (singular) as an alias to the existing --bootstrap-servers parameter in kafka-streams-application-reset.sh, deprecating the non-standard plural form. All other Kafka CLI tools use --bootstrap-server; the inconsistency in the streams reset tool causes user confusion.

TLDR: Adds end-to-end latency metrics for source connectors (transform-chain-source-record-time, convert-source-record-time) and sink connectors (record wall-clock lag, convert/transform time), providing a complete per-stage latency breakdown. Existing metrics only covered interaction with the external system (poll-batch-time, put-batch-latency), leaving the transformation and serialization pipeline unobservable.

TLDR: Optimizes KStream-KStream inner self-joins (where both sides read from the same topic) to use a single state store instead of two identical stores. Self-joins in Kafka Streams were treated as regular joins, creating two stores that always contained identical data, doubling state storage and write amplification unnecessarily.

TLDR: Adds a validateOnly option to the AlterPartitionReassignments AdminClient API that lets callers verify a reassignment request (including replication factor impact) without executing it, and adds a targetReplicationFactor field to prevent accidental RF changes. AdminClient#describeTopics returns the full intermediate replica set during ongoing reassignments, making it impossible for callers to reliably compute the current RF without a TOCTOU race condition.

TLDR: Adds metadata-apply-error-count and metadata-load-error-count metrics to broker metadata processing, and MetadataErrorCount to the controller, tracking failures during MetadataDelta generation and MetadataImage application in KRaft mode. Errors during metadata log processing can leave the node's in-memory state inconsistent with no observable signal, making diagnosis impossible without grepping logs.

TLDR: Extends KRaft mode to handle individual JBOD log directory failures by having the broker signal the controller via BrokerHeartbeat when a directory goes offline, triggering LeaderAndIsr updates for affected replicas. In ZooKeeper mode the broker directly notified the controller; in KRaft the broker's continued heartbeats masked per-directory failures, leaving partitions in that directory without leader failover.

TLDR: Adds a `KStream.recursively(UnaryOperator<KStream> op)` DSL operator that pipes a stream's output back into an earlier stage of the same topology, enabling iterative graph traversal and recursive algorithms natively. Recursive patterns currently require separate loopback topics and a second `KafkaStreams` instance, incurring extra serialization, latency, and operational complexity.

TLDR: Adds schema.namespace as a distinct parameter to the SetSchemaMetadata SMT, allowing the Avro schema namespace to be set independently from the schema name. Previously the only way to set a namespace was to embed it in schema.name as a fully qualified name, preventing users from reusing the original table/topic name while only changing the namespace.

TLDR: Introduces a separate `producer.id.expiration.ms` configuration distinct from `transactional.id.expiration.ms` so idempotent-only producers (without a `transactional.id`) have their producer IDs expired independently and on a shorter horizon. Previously both expiration windows were conflated, preventing fine-grained control over producer state cleanup for non-transactional producers.

TLDR: Introduces dynamic KRaft controller membership changes—adding and removing voters from the Raft quorum—without requiring all controllers to be shut down and manually updated. Before this KIP, changing the controller quorum composition required a full cluster shutdown and manual on-disk state manipulation, making controller node replacement and cluster scaling operationally risky and disruptive.

TLDR: Adds a RemoteLogMetadataManager#remoteLogSize(TopicIdPartition) API and a RemoteLogSizeBytes metric, allowing RLMM implementations to compute total remote tier size without listing all segment metadata. The existing O(num_remote_segments) scan via listRemoteLogSegments() becomes prohibitively expensive as the number of remote segments grows, slowing the retention enforcement loop.

TLDR: Adds a `requireStable` flag to `AdminClient.listConsumerGroupOffsets()` so callers can opt into seeing only stable (non-pending-transactional) committed offsets, consistent with the behavior available in `KafkaConsumer.committed()`. Previously the admin API always returned the latest committed offset regardless of in-flight transactions, making it impossible to observe only fully committed group state.

TLDR: Adds totalBytes and usableBytes fields to every logDirs entry in the kafka-log-dirs.sh JSON output (version 2), exposing log directory disk capacity alongside replica offset data. KIP-827 added this data to the broker API but the CLI tool did not surface it, forcing operators to separately inspect disk usage.

TLDR: Replaces the Classic stop-the-world consumer rebalance protocol with an incremental, server-side protocol where each consumer independently heartbeats and the broker's group coordinator manages assignment reconciliation without a synchronization barrier. The Classic protocol's full-group sync (all members stop, all rejoin) causes throughput disruption proportional to group size, and its thick client-side assignor design creates complex failure modes and slow recovery.

TLDR: KIP-847 adds a `ProducerIdCount` JMX metric to each broker that reports the number of active producer IDs tracked by `ProducerStateManager`. Operators had no lightweight way to monitor idempotent and transactional producer count; the only existing visibility (from KIP-360) required parsing log or snapshot data rather than querying a metric.

TLDR: Adds `Metrics.addMetricIfAbsent(MetricName, Measurable)` as an atomic get-or-create operation on the metrics registry. The existing two-step pattern (check existence, then `addMetric`) is not thread-safe: concurrent threads registering the same instance-level metric race and one throws `IllegalArgumentException`.

TLDR: Extends group offset reset with richer InitialOffsetResetStrategy options (e.g., EARLIEST_LOCAL, BY_DURATION) to handle scenarios such as consumer groups encountering newly added partitions after a topic partition count increase. The default auto.offset.reset=latest causes newly added partitions to start at the latest offset, silently skipping data that was produced into those partitions before the consumer group's rebalance detected the new partition count.

TLDR: Enforces in KRaft mode that fenced or in-controlled-shutdown replicas cannot join the ISR (new INELIGIBLE_REPLICA error on AlterPartition) or be elected leader, and persists the controlled-shutdown state in the metadata log. Without this enforcement, a leader could add a fenced replica to the ISR via a stale AlterPartition, allowing a broker with missing data to serve as leader and cause data loss.

TLDR: KIP-838 proposes a tool that reads a sample of segment log data and simulates the compression ratio that would be achieved if batching and compression were enabled, outputting the results as JSON. Cluster operators serving producers they do not control have no way to demonstrate concrete byte-reduction numbers to those producers before asking them to change their configuration.

TLDR: Exposes the `DescribeQuorum` API (KIP-595) through the `AdminClient` and adds voter lag computation in units of committed offset delta rather than raw log end offset. The API existed at the protocol level but was not accessible via the Admin client, and raw `LogEndOffset` comparison does not accurately reflect replication lag.

TLDR: Introduces a `NoOpRecord` written periodically to the `__cluster_metadata` log to advance the high-watermark, and exposes KRaft controller quorum health metrics so monitoring systems can verify that committed offsets are progressing. Without this, a quorum that has stalled (e.g. due to a missing majority) cannot be detected by watching offset advancement alone.

TLDR: Formally marks KRaft mode as production-ready for new clusters starting in Kafka 3.3 and deprecates ZooKeeper mode. This closes the multi-year KIP-500 effort by declaring the ZooKeeper dependency unnecessary for all production use cases covered by the new self-managed metadata quorum.

TLDR: Adds a `log-recovery-progress` metric (bytes recovered, total bytes, estimated completion percentage) to broker startup so operators can monitor log recovery progress. Log recovery on unclean shutdown is currently opaque: the broker logs nothing about progress, making it impossible to distinguish a slow recovery from a hung broker.

TLDR: Adds an `auto.include.jmx.reporter` configuration (deprecated in 3.x, removed in 4.0) to allow disabling the `JmxReporter` that was previously always injected regardless of the `metric.reporters` setting, and makes `JmxReporter` a standard entry in `metric.reporters` default in Kafka 4.0. Without this, operators in environments that do not use JMX cannot eliminate the JmxReporter overhead or consolidate metrics to a single explicitly configured reporter.

TLDR: Adds a `print.topic` property to `kafka-console-consumer` that prefixes each consumed message with the source topic name. When consuming from multiple topics via `--include`, there is no existing way to identify which topic a given record came from.

TLDR: Adds missing validators to configuration parameters across AdminClientConfig, ConsumerConfig, ProducerConfig, SaslConfigs, and broker configs (e.g., NonNullValidator for serializer/deserializer classes, valid-value checks for security.protocol, compression.type, ssl.client.auth). Without validators, invalid values for these configs are only discovered at runtime when the misconfigured component attempts to use them.

TLDR: KIP-827 adds `totalBytes` and `usableBytes` fields to the `DescribeLogDirs` API response (v4) and exposes them via `LogDirDescription` in the `AdminClient`. Storage metrics are available via JMX for trend monitoring and alerting, but automation and tooling that need a point-in-time disk capacity check (e.g. to verify a resize or confirm topic deletion freed space) previously had no Kafka-native API to query disk sizes.

TLDR: KIP-826 formally documents the set of operating systems, hardware architectures, and JVM versions that Apache Kafka officially supports and tests against. Without an explicit support matrix, users running Kafka on non-x86_64 platforms (e.g., aarch64, ppc64le) or with native dependencies like RocksDB and zstd have no authoritative guidance on whether their environment is supported.

TLDR: Adds an EmitStrategy API to TimeWindowedKStream and SessionWindowedKStream with ON_WINDOW_CLOSE (emit only the final result when the window closes) and ON_WINDOW_UPDATE (current behavior: emit for every record). The existing suppress(Suppressed.untilWindowCloses()) approach uses a redundant in-memory buffer with its own changelog topic, adding CPU, memory, disk, and operational overhead to achieve what is logically a windowing-level concern.

TLDR: Adds a `--max-batches` flag to `kafka-dump-log.sh` to limit the number of record batches printed when dumping segment logs. Without this, operators have no visibility into producer batching behavior in near-real-time without drowning in unbounded output, making it impossible to detect poor batching or compression configurations at scale.

TLDR: Extends `AdminClient.describeConfigs()` to accept an optional set of specific config key names within a `ConfigResource`, returning only those keys rather than all configurations. Currently the API always returns the entire config set for a resource, which is wasteful over large broker or topic configs when callers only need a small subset.

TLDR: Extends Kafka Connect Single Message Transforms (SMTs) to support nested field access and modification using dot-notation paths (e.g., `field.subfield`). Existing SMTs only operate on root-level fields, requiring custom SMT implementations for any transformation involving nested Struct or Map fields.

TLDR: Extends KStream with a process(ProcessorSupplier, Named, String...) method that integrates the new strongly-typed Processor API (KIP-478) directly into the DSL, replacing the legacy transform() and transformValues() methods with a unified processing operator. The existing Transformer-based operators were tied to the old ProcessorContext, lacked support for one-to-many output, and required multiple overlapping DSL methods to cover different combinations of value/key/context access.

TLDR: Adds `KStream.merge(Collection<KStream>)` overloads so multiple streams can be merged in a single DSL call instead of chaining repeated binary `merge()` calls. Merging N streams via repeated binary merges creates N-1 merge nodes in the topology graph, complicating debugging and monitoring.

TLDR: Fixes inconsistencies in the dynamic log-level adjustment API introduced by KIP-412, specifically adding support for `OFF` and `ALL` log levels and returning an error instead of silently falling back to `DEBUG` for unrecognized level strings. The JMX-based approach and the Admin API approach behave differently for these edge cases, making dynamic log management unreliable.

TLDR: Proposes stable topology identifiers so that adding, removing, or reordering subtopologies does not invalidate task IDs and local state stores for unchanged subtopologies. Currently any topology change that shifts topic-group ordinals invalidates all existing task directories, forcing a full state rebuild even for unaffected stores.

TLDR: Migrates `kafka-get-offsets.sh` from using `KafkaConsumer` to `AdminClient` and extends the tool to accept all `OffsetSpec` types (earliest, latest, max-timestamp, timestamp). The consumer-based implementation could not support the `max-timestamp` spec introduced in KIP-734 since that spec is only available via `AdminClient`.

TLDR: Extends the static membership protocol so that the group leader can return a `JOIN_GROUP` response indicating it should skip recomputing the partition assignment when it knows the assignment is unchanged. Currently, when a static leader rejoins the group, the coordinator silently designates it leader without informing it, causing the leader to send a stale assignment that wastes CPU and can trigger unnecessary rebalances.

TLDR: KIP-813 introduces shareable state stores in Kafka Streams, allowing multiple Streams applications to read from a single physical state store without each application maintaining its own changelog-backed replica. Today each application that needs the same state must independently replicate the full changelog topic, duplicating storage and reprocessing costs.

TLDR: Rate-limits the explicit log-truncation requests that Kafka Streams sends for repartition topics so they are not issued on every commit. When `commit.interval.ms` is very low (e.g. under `exactly_once_v2`) the resulting flood of `DeleteRecords` RPCs creates significant broker overhead.

TLDR: Adds a `null.marker` property to `kafka-console-producer.sh` that, when set, treats any matching input string as a null value, enabling production of tombstone records from the CLI. There is currently no way to produce null-valued records (tombstones) via the console producer, requiring users to rely on external tools like `kcat`.

TLDR: Extends the `TimestampConverter` SMT to support unix timestamp precision other than milliseconds—specifically seconds, microseconds, and nanoseconds—via a new `unix.precision` config. The existing SMT hard-codes millisecond precision for Unix Long types, breaking interoperability with external systems that produce or expect timestamps at different granularities.

TLDR: Unifies the disparate Kafka Streams metadata APIs (`StreamsMetadata`, `TaskMetadata`, `ThreadMetadata`, `KeyQueryMetadata`) into a single, consistent query interface. Users currently must call multiple different methods to gather state across metadata classes, with no single entry point to retrieve combined metadata.

TLDR: Adds `WindowKeyQuery` and `WindowRangeQuery` implementations of the `Query` interface from KIP-796 (Interactive Queries v2) for querying window and session stores by key and time range. IQv2 launched without session/window query support, meaning Streams applications using window/session stores could not use the new unified IQ API.

TLDR: Adds a `RangeQuery` implementation of the `Query` interface from KIP-796 (Interactive Queries v2) to support bounded range queries and full scans over key-value state stores. IQv2 launched without range/scan query support, leaving key-value stores accessible only via the deprecated point-lookup `KeyQuery`.

TLDR: KIP-804 tags the `OfflinePartitionsCount` controller metric with the topic name of the offline partition(s). The existing metric is an aggregate scalar with no topic label, preventing operators from building topic-scoped alerts, routing alerts to the right team, or filtering out noise from non-critical test topics.

TLDR: Exposes the task ID and connector name to `SourceTask` and `SinkTask` implementations via the `TaskContext` interface so connectors can use them for custom metrics, logging, or debugging. While KIP-449 added these values to the SLF4J MDC for log enrichment, there was no programmatic API for connectors to access them directly.

TLDR: Extends the Connect REST API (`PUT /connector-plugins/{name}/config/validate`) to also validate the configuration of SMTs, SMT predicates, and converters in addition to connector configurations. Currently there is no way to validate SMT or converter config before registering a connector—invalid configs are only discovered at connector startup.

TLDR: Introduces `StandardAuthorizer`, a new built-in KRaft-native authorizer that stores ACLs in the `__cluster_metadata` topic instead of ZooKeeper. In KRaft mode the existing `AclAuthorizer` still required a ZooKeeper instance, which defeated the goal of removing ZooKeeper entirely.

TLDR: Adds an optional `Reason` string field to `JoinGroupRequest` and `LeaveGroupRequest` so the broker can log why a consumer joined or left the group. Broker-side rebalance troubleshooting was hampered by the absence of a human-readable reason in these requests, since only the client side logged the reason.

TLDR: Fixes an inconsistency in `KafkaProducer` callback behavior where `Callback.onCompletion` passes a non-null placeholder `RecordMetadata` in some error paths but null in others, contrary to the Javadoc contract specifying that exactly one of `metadata` or `exception` is null. The inconsistency was introduced in PR #4188 (KAFKA-6180) and can cause NPEs or incorrect error handling in callback implementations.

TLDR: Adds parse.headers=true support to kafka-console-producer.sh's LineMessageReader, with configurable delimiters for header blocks, individual headers, and key-value pairs within a header. The console producer previously had no way to include record headers, limiting its usefulness for testing header-aware consumers and SMTs.

TLDR: KIP-796 redesigns the Kafka Streams Interactive Query (IQ) API to be generic and extensible, replacing the typed `QueryableStoreTypes` with a `StateQueryRequest`/`StateQueryResult` model that decouples query types from store implementations. The existing IQ API is tightly coupled to specific store implementations and cannot be extended by third-party store plugins without breaking changes.

TLDR: Replaces the `UniformStickyPartitioner` with a strictly uniform variant that switches the target partition based on the number of records sent rather than on batch creation events. The original sticky partitioner sent disproportionately more records to slower brokers because batches were created more frequently for them, creating a positive-feedback loop that worsened the slowdown.

TLDR: Adds originalTopic, originalKafkaPartition, and originalKafkaOffset fields to SinkRecord so that sink tasks overriding preCommit() can return the pre-transform offsets that the framework expects. When a topic-mutating SMT (e.g., RegexRouter) changes the topic name, the connector only sees the post-transform topic/partition/offset, causing preCommit() to return wrong offsets and breaking at-least-once offset tracking.

TLDR: Adds `recordMetadata()` to the `StateStoreContext` public interface so state stores can access the source record's offset and partition metadata during processing. `AbstractProcessorContext` already implements this method internally, but it is not exposed through the `StateStoreContext` interface, preventing stores from implementing Read-Your-Writes consistency or offset tracking.

TLDR: Allows `num.network.threads` to be configured per listener via the `listener.name.<NAME>.num.network.threads` notation so each listener's thread pool can be sized independently. A single global `num.network.threads` value is wasteful when listeners carry very different traffic volumes (e.g. inter-broker vs. external client listeners).

TLDR: Adds JMX metrics that expose the configured quota limits (bytes/sec for produce/fetch and request quota) tagged at the client-id/user granularity alongside the existing throttle-time metrics. Without the quota ceiling value, operators cannot perform capacity planning or set meaningful alerts because they have no way to know how close clients are to being throttled.

TLDR: KIP-785 adds automatic storage formatting to KRaft brokers on first startup when storage has not already been formatted, eliminating the need to run `kafka-storage.sh format` as a separate step. The manual formatting step adds operational friction to KRaft deployments, especially in containerized and automated provisioning environments.

TLDR: Adds a top-level `errorCode` field to the `DescribeLogDirsResponse` so authorization failures (e.g., missing `Describe` on the Cluster resource) are surfaced explicitly rather than returning an empty response. Currently, an empty response is ambiguous—clients cannot distinguish a legitimate empty result from an authorization failure or an unknown server error.

TLDR: Standardizes exception handling in Kafka Streams by ensuring all exceptions propagated to the `StreamsUncaughtExceptionHandler` are consistently wrapped as `StreamsException` with a well-defined subtype hierarchy. Previously only some exceptions were wrapped, making it impossible to write reliable handler logic or reason about the type of failure.

TLDR: Introduces a `batch.max.bytes` producer config that controls the maximum size at which a batch is drained from the accumulator, decoupling it from `batch.size` which governs memory allocation. Currently `batch.size` serves both roles, forcing operators to choose between memory waste (large allocation) and low throughput (small batch), with no way to tune the two concerns independently.

TLDR: Fixes MirrorMaker 2 client configuration propagation so that replication-level configs such as `A→B.producer.batch.size` are correctly applied to the respective producer, consumer, and admin clients. The existing MM2 configuration parsing silently ignored any client-specific overrides that were not common connection properties.

TLDR: Adds per-codec fine-grained compression configuration options (e.g., compression level, strategy, checksum mode) to producer, broker, and topic configs, going beyond the single compression.level parameter. Kafka allowed only default codec parameters with limited level tuning, preventing operators from trading off compression ratio vs. speed at the topic level for use cases like real-time ingest vs. long-term retention.

TLDR: Adds a new `SourceTask.commitRecord(SourceRecord, RecordMetadata)` callback and error handling hook so source connectors can handle producer exceptions (e.g., record-too-large) per record rather than having the task killed. Previously, any producer write failure from a source task immediately killed the connector with no opportunity for the connector to log, route to a DLQ, or skip the bad record.

TLDR: KIP-778 defines the rolling upgrade and downgrade protocol for Kafka clusters running in KRaft mode, gating new RPC versions and metadata record formats behind a cluster-wide metadata version (KV) that is only bumped once all nodes are upgraded. Without this mechanism, mixed-version KRaft clusters could produce or process record formats incompatible with older nodes, making safe rolling upgrades impossible without a disruptive "double roll".

TLDR: Establishes a consistent convention for `AdminClient` `*Result` class constructors—package-private by default, with static factory methods for test mocking—to make mock-based testing easier without exposing internal implementation details. Previous KIPs proposed either making all constructors public (risking ABI breakage) or keeping them package-private (blocking testability), leaving users unable to write reliable mock Admin tests.

TLDR: Enables Kafka Streams foreign-key (FK) table joins to work correctly when the primary or foreign-key tables use custom partitioners by making the subscription and response topics copartition with the relevant tables. Previously FK joins silently produced missing results if either table was partitioned with a non-default partitioner, because lookups could be routed to Streams instances that did not hold the required state.

TLDR: Makes all `*Result` class constructors in `AdminClient` package-private, resolving the inconsistency where most had package-private constructors but some were accidentally made public. Public constructors on result classes caused a breaking API change during Kafka 3.0 development when signatures were changed without noticing a constructor was public.

TLDR: Renames the producer metrics `bufferpool-wait-time-total`, `io-waittime-total`, and `iotime-total` to include `-ns` or `-ms` suffixes (e.g., `bufferpool-wait-time-ns-total`) to make the unit explicit and consistent with the rest of the metric family. These three metrics deviate from the established naming convention where nanosecond metrics include `-ns` in their name, causing confusion about units.

TLDR: KIP-769 adds a `connectorsOnly=false` query parameter to `GET /connector-plugins` so it lists all installed plugin types (connectors, SMTs, converters, header converters, predicates), and adds a new `GET /connector-plugins/{plugin}/config` endpoint returning the plugin's configuration definition. Previously, only connector plugins were discoverable via the REST API; other plugin types had to be found by inspecting the Connect worker's classpath or documentation.

TLDR: KIP-767 introduces per-record latency metrics (average and max) to Kafka Connect that measure the delay between a record's source timestamp and its successful delivery to the sink. Connect currently provides no common latency instrumentation, forcing operators to implement ad-hoc monitoring outside the framework.

TLDR: Adds a configurable connection-acceptance rate limit to the Kafka broker's `Acceptor` so that a burst of new TCP connections (e.g. after a preferred-leader election) does not fill the SYN backlog and drop subsequent connection attempts. Under high partition counts a rolling restart or leader election causes hundreds of clients to reconnect simultaneously, overwhelming the kernel TCP backlog.

TLDR: Adds unbounded (null-terminated) range and reverse-range queries to Kafka Streams `ReadOnlyKeyValueStore` so callers can scan from a lower bound to the end of the store, or from an upper bound to the beginning. The existing `range(K from, K to)` required both bounds, forcing workarounds for open-ended scans.

TLDR: Extends Kafka Connect to automatically delete records from source topics once all SinkConnectors that subscribe to those topics have processed them, enabling low-latency storage-efficient pipelines. Without this, Connect pipelines that use Kafka purely as a transport medium (not for long-term retention) must either over-provision retention or risk SinkConnectors falling behind a short retention window.

TLDR: Introduces a new `blocked-time-total` metric in Kafka Streams that accumulates the total wall-clock time a stream thread spends blocked waiting on Kafka (poll, commit, flush) rather than processing records. Existing metrics like `poll-ratio` and `processing-ratio` are instantaneous ratios from a single poll loop iteration and are too coarse-grained to identify systemic blocking bottlenecks.

TLDR: Introduces broker-level `min.topic.segment.ms` and `min.topic.segment.bytes` configurations that enforce a minimum floor on the corresponding topic-level settings. Small values for `segment.ms` or `segment.bytes` cause an explosion of log segment files per partition, which can exhaust file descriptors or memory and crash brokers.

TLDR: Adds a `repartitionRequired` flag to Kafka Streams' internal graph optimization so that a `selectKey`/`map` followed by a `groupByKey` does not insert an unnecessary repartition topic when the upstream stream is already partitioned by the new key. Every key-changing operation currently forces a repartition before any stateful aggregation, even when the resulting key is identical to the existing partition key, wasting network bandwidth and adding end-to-end latency.

TLDR: Introduces a new `AUTO_CREATE` ACL operation that grants permission specifically for auto-topic creation (triggered by `MetadataRequest`) distinct from the `CREATE` ACL used for explicit `CreateTopicsRequest`. Currently a single `CREATE` ACL covers both auto-creation and explicit creation, preventing administrators from allowing auto-creation (which uses broker defaults) without also granting full topic creation rights.

TLDR: Makes Scala case classes in the Kafka codebase `final` to align with Scala best practices for algebraic data types (ADTs) that represent immutable data. Non-final case classes break pattern-match exhaustiveness checking and allow subclassing that can violate the value-semantics contract of case classes.

TLDR: Adds `--files` and `--file-separator` options to `kafka-console-producer.sh` to allow bulk-producing records from one or more files in a single invocation. Currently the console producer can only read from stdin or a single file via shell redirection, requiring complex shell scripting to ingest multiple log files.

TLDR: Introduces two new KRaft controller metrics: `ActiveBrokerCount` (registered, unfenced brokers) and `FencedBrokerCount` (brokers that registered but are fenced). Without these metrics, a broker that is alive but failing to emit metrics or logs is indistinguishable from a broker that never registered, making cluster health assessment unreliable.

TLDR: Revises the KRaft metadata record schemas introduced in KIP-631 by adding flexible versions, missing fields (e.g. for KIP-455 reassignment), and proper `entityType` annotations. Several record types were incorrectly defined without flexible versions, and some lacked fields required by subsequent KIPs.

TLDR: Adds a `ConnectorClientConfigRequest` and corresponding `ConnectorClientConfigOverridePolicy` extension point so Connect connector implementations can declare recommended client config overrides for their embedded producers, consumers, and admin clients. There was no structured way for connectors to influence the client configuration used by the Connect framework for their tasks.

TLDR: Converts `TaskMetadata` and `ThreadMetadata` from concrete classes to interfaces with internal implementations, separating the public API contract from the implementation details. Users should never instantiate these metadata objects directly—making them concrete classes with public constructors prevents the Streams team from safely evolving the internal representation.

TLDR: KIP-743 removes the `0.10.0-2.4` config value for `built.in.metrics.version` in Kafka Streams (Kafka 3.0), leaving `latest` as the only valid value. The legacy metric structure introduced by KIP-444's predecessor was effectively deprecated when `latest` became the default in 2.5, and maintaining dual metric code paths is error-prone; new metrics added post-KIP-444 only exist in the `latest` structure, meaning users on `0.10.0-2.4` already receive a mix of old and new metric formats.

TLDR: Changes `TaskMetadata.taskId()` to return a `TaskId` object instead of its `String` encoding so callers receive a strongly-typed value with direct access to sub-topology and partition fields. The `String` representation required callers to parse the encoding themselves, and made it impossible to add new `TaskId` fields without breaking existing parsing code.

TLDR: Makes the deprecated `internal.key.converter` and `internal.value.converter` Connect worker properties fatal on startup so workers immediately fail rather than silently producing corrupted internal topic data. Workers configured with a non-default internal converter cannot read internal topic data written by workers with a different converter, leading to hard-to-diagnose data corruption.

TLDR: KIP-737 adds a `canTrackSource()` method to the `ReplicationPolicy` interface in MirrorMaker 2, allowing implementations to declare whether they can reverse-map a topic name back to its source cluster. Without this capability, the `IdentityReplicationPolicy` (needed for MirrorMaker 1 migration) cannot coexist cleanly with MM2's offset syncing logic, which assumes topic names encode source cluster identity.

TLDR: Adds per-follower latency breakdown metrics to the fetch purgatory so operators can see how long fetch requests spend waiting for each follower to acknowledge, decomposing end-to-end produce latency more precisely. The existing `RemoteTime` metric for produce requests aggregates all follower wait time without distinguishing between network congestion and slow fetch processing per follower.

TLDR: Adds a per-consumer-group `session.timeout.ms` override capability so that individual groups can be configured with session timeouts outside the broker's global `[group.min/max.session.timeout.ms]` range. In multi-tenant cloud environments workloads have highly variable GC and network characteristics, making a single global timeout range inadequate for all groups.

TLDR: Adds OffsetSpec.MaxTimestampSpec to AdminClient.listOffsets(), returning the offset and timestamp of the record carrying the highest message timestamp in a partition. Provides a lightweight liveness indicator for partitions without requiring a consumer or metric scrape, noting that MM2 timestamp preservation means this may not equal the latest append time.

TLDR: Changes the default value of `replication.factor` in Kafka Streams from `1` to `-1` (use broker default) to align with the broker-side default replication factor. The hardcoded default of `1` is unsafe for production—users must remember to override it—while `-1` (available since Kafka 2.4, KIP-464) delegates to the broker's `default.replication.factor`.