CodePark/ questions / Monitoring, lag & scaling consumers
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › Monitoring, lag & scaling consumers

What is consumer lag, how do you measure it in Kafka, and why is it the key health metric for a consumer?

$ kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
    --describe --group billing
TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
orders  0          120400          120410          10
orders  1          98012           131870          33858

Consumer lag is how far a consumer group is behind the newest data.

Kafka computes it per partition:

  • Take the log-end offset. This is the offset of the next message to be written.
  • Subtract the group's committed offset. This is the last position the group has saved as done.
  • Add up the result for all partitions to get the group's lag.

The command kafka-consumer-groups.sh --describe shows it. Exporters send it to Prometheus or a similar monitoring tool.

It is the key metric because it shows whether consumers keep up, whatever the cause:

  • slow code
  • a stuck partition
  • a crashed consumer
  • a traffic spike

If lag grows steadily, the group is falling behind. It will never catch up unless you act.

Look at the trend and at each partition's value, not one total number. One partition with huge lag, like partition 1 above, points to a hot key (one key that gets far more messages than others) or a stuck consumer. Where you can, alert on lag in time: how old the oldest unprocessed record is. Time maps directly to how long users wait.

  • Lag is the log-end offset minus the committed offset, per partition.
  • You measure it with kafka-consumer-groups or an exporter.
  • Lag shows whether consumers keep up, whatever the cause.
  • Watch the trend and look for one partition with much more lag than the others.
  • Lag measured in time maps best to the impact on users.