CodePark/ questions / Queues vs streams vs pub/sub
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › Queues vs streams vs pub/sub

Kafka consumers pull messages, while RabbitMQ pushes messages to subscribed consumers. How does each model handle a slow consumer?

In Kafka, a slow consumer just reads less often, and unread records wait in the log. In RabbitMQ, you need a prefetch limit, or the broker floods a slow consumer.

Kafka (pull): the consumer calls poll() when it is ready for more records. A slow consumer simply polls less often. Unread records stay in the log. The cost shows up as consumer lag (how far the consumer is behind the newest record). It does not put memory pressure on the broker or the consumer. One caveat: the consumer must still call poll() often enough, or Kafka removes it from the consumer group.

RabbitMQ (push): the broker pushes messages to consumers that registered with basic.consume. Without a limit, the broker sends as fast as it can. A slow consumer can then hold thousands of unacked messages in memory. The fix is a prefetch limit, set with basic.qos. The broker stops sending once the consumer has that many unacked messages. This gives you backpressure: the slow side controls the speed. Messages the consumer has not taken stay in the queue, and other consumers can get them.

  • In Kafka, the consumer pulls with poll(), and slowness shows up as lag.
  • In RabbitMQ, the broker pushes messages to consumers.
  • Without a prefetch limit, a slow RabbitMQ consumer holds too many messages in memory.
  • A basic.qos prefetch limit caps unacked messages and gives backpressure.