What is a poison message, and how should a consumer handle failures so one bad message doesn't take the system down?

A poison message is a message that fails every time it is processed. Common causes:

• invalid JSON
• an unknown schema version
• a reference to data that does not exist
• input that triggers a bug

If you retry it forever, it causes damage:

• In Kafka, it blocks its partition.
• In RabbitMQ, it loops in the queue.
• Consumers may crash again and again.

First, classify the error.

• Transient failures are temporary, such as timeouts, a downstream 503, or a deadlock. Retry them with exponential backoff (double the wait each time) and jitter (a small random extra wait).
• Permanent failures will never succeed, such as validation errors, deserialization errors, or business rule violations. Send them to the dead-letter queue (DLQ) at once. A DLQ is a separate queue that holds failed messages for later inspection.

Then protect the system:

• Cap retries with a maximum attempt count.
• Send messages that still fail to the DLQ, together with the error, stack trace, attempt count and source.
• Alert on DLQ depth (how many messages are in it).
• Investigate, fix the cause, then redrive the messages (send them back to be processed again).
• Catch errors per message, so one failure does not crash the whole consumer.