Explain at-most-once, at-least-once and "exactly-once" delivery. How do you get effectively-once processing in practice?

These three terms describe how many times a message can be delivered:

• At-most-once: the consumer acknowledges the message before processing it. A crash during processing loses the message, but it is never processed twice.
• At-least-once: the consumer acknowledges only after processing. A crash before the ack causes redelivery, so the message may be processed twice. This is the usual default.
• Exactly-once: each message is delivered once and only once.

True exactly-once delivery over a network is not possible in general. The reason: the sender cannot tell a lost message apart from a lost ack. Some systems offer exactly-once semantics inside their own boundary. For example, Kafka transactions give exactly-once for read-process-write between Kafka topics. But side effects in external systems, like a database or an email service, still need care.

In practice, you use at-least-once delivery plus an idempotent consumer. Idempotent means processing the same message twice has the same effect as processing it once. To do this:

• Give each message a unique id.
• Record the processed id in the same transaction as the side effect. Or use a unique constraint or an upsert.
• Skip any message whose id you have already seen.

The result is effectively-once processing.