What triggers a consumer group rebalance, why are rebalances painful, and how do you reduce the pain?

A rebalance gives the partitions of a topic to the consumers of a group again. It happens when the group's members or its subscription change. Triggers:

• A consumer joins or leaves the group.
• A consumer misses heartbeats for longer than session.timeout.ms. A heartbeat is a small "I am alive" message the consumer sends.
• A consumer waits longer than max.poll.interval.ms between two poll() calls.
• Someone adds partitions to the topic.

Why it hurts: with the classic eager protocol, every member gives up all its partitions. Processing stops until the new assignment is in place. Work on records whose offsets are not committed yet is done again by another consumer. So you get pauses and duplicate processing. A rolling deploy of 10 pods can trigger a rebalance storm: many rebalances, one after another.

How to reduce the pain:

• Use the cooperative-sticky assignor. It moves only the partitions that need to move. The other consumers keep reading.
• Set group.instance.id. This is static membership: a restarting pod gets its partitions back without a rebalance, if it returns within the session timeout.
• Commit offsets in the partition-revoked callback, so the next owner starts from the right place.
• Keep the poll loop fast.
• Shut down cleanly, so the consumer leaves the group properly.

Kafka 4.0 also offers the newer consumer group protocol. In it, the broker (the Kafka server) drives incremental assignment.