CodePark/ questions / Kafka: topics, partitions & brokers
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › Kafka: topics, partitions & brokers

A keyed topic has 6 partitions and you increase it to 12 because consumers cannot keep up. What happens to existing data and to per-key ordering?

Existing records stay where they are, but new records for many keys now hash to a different partition, so ordering for those keys can break around the change

Existing records stay where they are, but new records for many keys now go to a different partition. Kafka never moves existing records between partitions. The default partitioner is simply hash modulo partition count. When the count changes from 6 to 12, many keys map to a new partition. Old records and new records for one key can then be read by different consumers, so they can be processed out of order.

"Kafka moves existing records so each key's history sits in its new partition" is wrong, because Kafka never moves data between partitions. "Kafka remembers each key's original partition" is also wrong: Kafka keeps no memory of where each key went before.


Plan partition counts at the start. You also cannot reduce the partition count later.