CodePark/ questions / Monitoring, lag & scaling consumers
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › Monitoring, lag & scaling consumers

Why is SNS combined with SQS ("fan-out") a common pattern on AWS, rather than having services subscribe to SNS directly over HTTP?

Because an SQS queue stores messages, and SNS does not. SNS is push-based pub/sub. It delivers each message to every subscription. But it does not keep messages for slow or offline consumers, apart from its own delivery retries. If an HTTP subscriber is down long enough, its messages are lost, unless you set up a dead-letter queue for that subscription.

With fan-out, each consuming service subscribes its own SQS queue to the SNS topic. The queue is a durable buffer for that service. Each service can:

  • consume at its own speed
  • be down for a while without losing messages
  • scale its workers on its own
  • use the queue's own retries, visibility timeout and DLQ (dead-letter queue)

One slow service no longer affects the others.

Two extra features help:

  • Subscription filter policies let each queue receive only the message types it cares about.
  • Raw message delivery sends the plain message body, without wrapping it in the SNS envelope.

This is the AWS version of a RabbitMQ fanout or topic exchange with one queue per service.

  • SNS pushes messages and does not keep them for offline consumers.
  • One SQS queue per service gives each service its own durable buffer.
  • Each service scales, retries and dead-letters on its own.
  • Filter policies send each queue only the messages it needs.
  • Raw message delivery removes the SNS envelope.