CodePark/ questions / Ordering, partitioning & concurrency
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › Ordering, partitioning & concurrency

How do SQS FIFO queues provide ordering and deduplication, and how do you still get parallelism from them?

A FIFO (first in, first out) queue keeps order per message group id. Messages with the same MessageGroupId arrive in the order they were sent. While one message from a group is in flight (received but not yet deleted), the queue does not give out later messages from that group. Different groups are independent. Different consumers can process them in parallel.

Deduplication works inside a 5-minute window. You can set a MessageDeduplicationId yourself. Or you can turn on content-based deduplication, so SQS uses a hash of the message body as the id. If a send uses an id already seen in that window, SQS accepts it but does not deliver it again.

To get parallelism, choose the group id like a Kafka key. Use the entity whose order matters, such as the order id. If you use one constant group id, the whole queue becomes sequential. The dedup window only protects against producer retries. So consumers still need to be idempotent: processing the same message twice must give the same result as processing it once.

  • Ordering is kept per MessageGroupId.
  • A message in flight blocks later messages of the same group.
  • Different groups are processed in parallel.
  • Deduplication uses a MessageDeduplicationId or a content hash, within a 5-minute window.
  • One constant group id makes everything sequential, and consumers still need to be idempotent.