CodePark/ questions / Monitoring, lag & scaling consumers
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › Monitoring, lag & scaling consumers

A topic has 12 partitions and the consumer group already runs 12 instances. Lag keeps growing during peak hours. What should you do?

Make each consumer faster (batch writes, concurrent processing within a partition where ordering allows) or add partitions

Make each consumer faster or add partitions. With 12 partitions, at most 12 consumers in a group get work. So "Scale the group to 24 instances" is wrong: instances 13 to 24 would sit idle. More throughput must come from faster processing per partition or from more partitions. Note that adding partitions changes which partition each key goes to. "Restart the consumers to trigger a rebalance" only adds a pause while the group rebalances. "Increase the topic's retention so no data is lost" prevents data loss, but it does not reduce lag.