What is log compaction in Kafka, how do you delete a key from a compacted topic, and when would you use one?

Log compaction means Kafka keeps at least the latest record for every key, and removes older records for the same key in the background.

Normal topics use cleanup.policy=delete. Kafka removes old segments after retention.ms (a time limit) or retention.bytes (a size limit). A segment is one file of the partition's log. With cleanup.policy=compact, Kafka keeps the latest value per key instead.

Some details:

• Offsets do not change, so a compacted partition has gaps in its offsets.
• Kafka does not compact the active segment (the file it is still writing to). So recent duplicates stay for a while.

To delete a key, write a tombstone. A tombstone is a record with that key and a null value. Compaction later drops the older values for the key. Kafka removes the tombstone itself after delete.retention.ms.

When to use it: compacted topics suit current-state data, where only the latest value per key matters. Examples:

• a user profile or a product price per id
• configuration
• a changelog that a new service reads from the start to rebuild a table

Kafka uses a compacted topic itself for __consumer_offsets.