CodePark/ questions / Delivery guarantees & idempotent consumers
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › Delivery guarantees & idempotent consumers

A Kafka consumer charges a card for each PaymentRequested event and commits the offset after processing. It crashes after the charge succeeds but before the commit. What happens after it restarts?

The event is redelivered and the card is charged again, unless the handler is idempotent

The event is delivered again, so the card is charged twice unless the handler is idempotent. The consumer never committed the offset. The offset is the consumer's saved position in the partition. It still points at this event, so the next owner of the partition reads it again. This is at-least-once delivery: no event is lost, but duplicates can happen.

So the charge must be deduplicated. For example, send an
idempotency key to the payment provider, so the provider ignores a repeated charge.

"Kafka's idempotent producer prevents the second charge" is wrong. The idempotent producer only removes duplicates when the producer retries a write into Kafka. It does nothing about side effects on the consumer side, like charging a card.