An older producer config disables idempotence but keeps retries on and max.in.flight.requests.per.connection=5. Records for one key sometimes arrive out of order. Why, and what is the fix?

Retries reorder batches when several requests are in flight, and enabling idempotence fixes it. In flight means sent but not yet acknowledged by the broker. Say batch 1 fails and is retried, but batch 2, sent later, succeeds first. Then batch 2 is written before batch 1.

The idempotent producer adds sequence numbers to each batch. The broker uses them to reject batches that arrive out of order. So order holds with up to 5 requests in flight.

"Retries always reorder records; set retries=0" is wrong. Retries only reorder when several requests are in flight. Setting retries=0 keeps order only by giving up on temporary failures. That trades reordering for message loss.