A queue gives each message to one consumer. Pub/sub gives a copy to every subscriber. A stream keeps messages in a log that many readers can read and re-read.
A queue holds messages until a consumer takes them. Each message goes to one consumer. Once the consumer acknowledges it (says "done"), the queue deletes it. Several consumers on the same queue compete for the work. This is how you spread jobs across workers.
Pub/sub (publish/subscribe) delivers a copy of each message to every subscriber. Plain pub/sub, like Redis Pub/Sub, stores nothing. So a subscriber that is offline misses messages. Brokers like RabbitMQ or SNS give you durable fan-out by giving each subscriber its own queue. Fan-out means one message goes to many receivers. Durable means messages survive while a subscriber is offline.
A stream is an append-only log. New records are only added at the end. Reading does not delete anything. Each consumer group tracks its own position in the log, called the offset. The broker keeps data for a set time, called the retention period. Because of this:
- many independent groups can read the same data
- a group can rewind and replay old data
Kafka, RabbitMQ streams and Redis Streams work this way.
- A queue gives each message to one consumer, deletes it after the ack, and lets consumers compete for work.
- Pub/sub gives every subscriber a copy. Plain pub/sub keeps nothing for subscribers that are offline.
- A stream is an append-only log. Consumers track their offsets, and data is kept for a retention period.
- Streams allow replay and let many independent readers read the same data.
- Kafka introduction (https://kafka.apache.org/intro)kafka.apache.org
- Messaging patterns (https://www.enterpriseintegrationpatterns.com/patterns/messaging/)enterpriseintegrationpatterns.com