CodeParkA consumer group saves its progress as a committed offset for each partition. Kafka stores these offsets in the internal __consumer_offsets topic. The committed offset is the position of the next record to read. After a restart or a rebalance (Kafka moving partitions between consumers), the new owner of the partition continues from that offset.
Auto-commit: with enable.auto.commit=true, the client commits on a timer (auto.commit.interval.ms). The client knows which records it fetched. It does not know which records your code finished processing. This causes two problems:
- Lost messages: if you pass records to async work, the client can commit before processing finishes. If the app crashes, those records are lost.
- Duplicates: if the app crashes after processing but before the next commit, Kafka sends those records again.
Manual commit after processing: you call commitSync or commitAsync after your code finishes. This gives at-least-once delivery. Nothing is skipped. But a crash after processing and before the commit causes the records to be delivered again. So your handlers must be idempotent: running them twice gives the same result as running them once.
Manual commit before processing gives at-most-once delivery. A crash can lose records, but none are processed twice.
For better throughput, commit once per batch, not once per record.
- Committed offsets live in
__consumer_offsets, one per group and partition. - Auto-commit runs on a timer and does not know if processing succeeded.
- Auto-commit with async processing can lose messages, and crashes can cause duplicates.
- Committing after processing gives at-least-once delivery, so handlers must be idempotent.
- Committing before processing gives at-most-once delivery.