Kafka advertises exactly-once semantics. What exactly does it cover, and where does it stop?

Kafka's exactly-once covers read-process-write inside Kafka. It stops at Kafka's boundary.

How it works inside Kafka:

• A transactional producer is a producer with a transactional.id. It writes output records to one or more topics.
• It adds the consumer's offsets to the same transaction with sendOffsetsToTransaction. An offset is the consumer's saved position in a partition.
• Either all of it commits, or none of it does.
• Downstream consumers must use isolation.level=read_committed. Then they skip records from aborted transactions.

Kafka Streams uses this when you turn on exactly-once processing.

Where it stops: your handler may write to Postgres, call a payment API or send an email. Kafka cannot roll that back. Imagine the app crashes after the side effect but before the transaction commits. Kafka replays the record, and the side effect happens again.

For external systems, you still need at-least-once delivery plus idempotent processing. Idempotent means doing it twice has the same result as doing it once. Ways to do this:

• Dedupe by message id.
• Use upserts.
• Use idempotency keys.
• Store offsets in the same database transaction as the results.