How does a Kafka producer decide which partition a record goes to, with and without a key?

The producer picks the partition in this order:

• If you set the partition yourself, Kafka uses it.
• If the record has a key, Kafka hashes the key and picks a partition from the hash.
• If there is no key, the producer uses sticky partitioning.

With a key: the default partitioner (the code that chooses the partition) hashes the serialized key. The Java client uses the murmur2 hash. It then takes the hash modulo the number of partitions. So the same key always goes to the same partition, as long as the partition count does not change. This is what gives you ordering per key. For example, all events for user 42 stay in order.

Without a key: modern clients use sticky partitioning. The producer fills one batch for one partition, then moves to another partition. This gives bigger batches and better throughput than sending each record to the next partition in turn. Over time, the load is still spread across all partitions.

You can also plug in your own custom partitioner. Choose the key carefully either way. Some keys have few different values, or one value that is very common (a "hot" key). These keys put too much data on a few partitions, so the partitions become unbalanced.