What problem does the idempotent producer solve, how does it work, and what does it NOT protect against?

The idempotent producer stops retries from creating duplicate or reordered records.

The problem: a producer sends a batch, but the confirmation (the ack) is lost, for example because of a timeout or a short network problem. So the producer sends the batch again. Without idempotence, the broker may write the batch twice. This creates duplicates. With several requests in flight at once, a retry can also change the order of records.

How it works: with enable.idempotence=true, the broker gives the producer an id. The producer numbers each batch per partition with a sequence number. The broker then:

• drops a batch with a sequence number it has already written;
• rejects a batch that leaves a gap in the numbers.

So retries cannot duplicate or reorder records.

It needs:

• acks=all
• retries turned on
• at most 5 in-flight requests (max.in.flight.requests.per.connection <= 5)

Since Kafka 3.0, it is on by default.

What it does not cover: it only protects retries inside one producer session, per partition. Kafka sees new records in these cases:

• Your app crashes, restarts and sends the same business event again.
• Two app instances publish the same event.

For those, you need transactions or idempotent consumers. An idempotent consumer gives the same result if it handles a message twice.