One huge customer produces 40% of all events, and the Kafka topic is keyed by customerId. One partition's consumer lags hours behind while the others are idle. Why doesn't adding consumers help, and what can you do?

Adding consumers doesn't help because one partition can only be read by one consumer in a group. Kafka hashes each message key to pick a partition. So all events with the same customerId go to the same partition. That one partition, and the single consumer reading it, gets 40% of the load. This stays true no matter how many consumers you add.

First, check what really needs to stay in order. Maybe you need order per customer. Or maybe you only need it for something smaller, such as per order or per account. If so, use that smaller id as the key. The big customer's events then spread across many partitions, and the order that matters is kept.

If you really need order per customer, make that one consumer faster:

• Process different sub-entities (for example, different orders) at the same time, but keep each one in sequence.
• Batch database writes.
• Remove slow calls.

Other options:

• Send the big customer to its own topic, with more partitions and a dedicated consumer.
• Salt its key. Salting means adding a suffix, such as customerId-0 to customerId-7, so its events spread over 8 partitions. The cost: order now holds only within each salt, not across the whole customer.