CodeParkScale on the backlog (the work still waiting in the queue), not on CPU. Consumers that mostly wait on I/O (network or database calls) can fall far behind while their CPU stays low.
Which metric to use:
- Kafka: consumer lag. This is how many messages the consumer has not read yet.
- RabbitMQ: queue depth, or the number of ready messages.
- SQS:
ApproximateNumberOfMessagesVisible, or the age of the oldest message.
Tools such as KEDA (Kubernetes Event-Driven Autoscaling) scale Kubernetes deployments on these metrics.
Limits: In Kafka, each partition goes to only one consumer in a group. So consumers beyond the partition count sit idle. More importantly, the downstream system is usually the real bottleneck. For example, doubling the consumers doubles the database connections and writes. This can take the database down. To protect it:
- Set a maximum number of replicas.
- Limit concurrency per consumer, and limit prefetch (how many messages a consumer takes at once).
- Treat the queue as the buffer that absorbs peaks. Messages can wait there safely.
Avoid flapping (scaling up and down again and again). In Kafka, every scale event causes a rebalance: the group pauses and reassigns partitions to consumers. To reduce the cost:
- Use cooldowns between scale events.
- Use cooperative-sticky assignment or static membership, so fewer partitions move.
- Use a graceful shutdown. The consumer finishes its in-flight work and commits its offsets before it exits.
- Scale on lag, queue depth or oldest message age, not on CPU.
- In Kafka, useful consumers are capped at the partition count.
- Downstream capacity is often the real limit, so cap the number of replicas.
- Prefetch and concurrency limits give backpressure, so consumers don't take more than they can handle.
- Use cooldowns, cooperative rebalancing and graceful shutdown to avoid flapping.