CodeParkA dead letter queue (DLQ) is a separate queue for messages that could not be processed. A message goes there when:
- it fails processing a set number of times
- it expires
- it is rejected
Why do you need one? Some messages always fail, for example because their data is broken. This is called a poison message. Without a DLQ, the system would retry it forever and waste resources. In an ordered queue, it would also block every message behind it.
Most brokers support this directly. A broker is the server that stores and delivers the messages.
- RabbitMQ has dead letter exchanges.
- Amazon SQS has a redrive policy with a maximum receive count.
- BullMQ keeps a job in a failed state after all its attempts are used up.
Kafka has no built-in DLQ. So consumers usually publish failed messages to a separate topic.
To operate a DLQ well:
- Send an alert when messages land there.
- Store the error and the original metadata with each message.
- Fix the cause of the failure.
- Then redrive the messages. This means you move them back to the main queue to be processed again.
A DLQ that nobody watches is just silent data loss.
- A DLQ holds messages that failed N times or expired.
- It stops poison messages from looping forever or blocking other messages.
- Brokers support it: RabbitMQ has DLX, SQS has redrive, BullMQ has failed jobs.
- Alert when messages arrive, and keep the error context with each one.
- Fix the cause first, then redrive the messages to the main queue.
Read more