A consumer fetches up to 500 records per poll and each one takes about 1 second to process. Logs show the same records being processed again and again, and the group keeps rebalancing. Explain and fix it.

One batch takes longer than max.poll.interval.ms, so Kafka thinks the consumer is dead.

What happens, step by step:

• One poll() returns up to max.poll.records records. The default is 500.
• At 1 second each, one batch takes about 500 seconds.
• max.poll.interval.ms defaults to 300000 ms (5 minutes). The batch takes longer than that.
• The consumer does not call poll() again in time, so it is considered failed and leaves the group.
• A rebalance moves its partitions to another member. That member starts from the last committed offset, so it processes the same records again.
• The original consumer's later commit is rejected. Then the loop repeats.

Fixes:

• Lower max.poll.records, so a batch finishes well within the interval.
• Raise max.poll.interval.ms if processing really is slow.
• Make the work itself faster: batch the database writes, and run I/O in parallel inside the batch.
• For rare long tasks, call pause() on the partitions and keep calling poll(), so the consumer stays in the group.

Also make handlers idempotent: processing the same record twice must give the same result as once. Some replay will still happen.