CodePark/ questions / Kafka: topics, partitions & brokers
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › Kafka: topics, partitions & brokers

Explain Kafka's core building blocks: topic, partition, offset, broker and replica.

A topic is a named stream, split into partitions. Offsets mark positions in a partition. Brokers are the servers, and replicas are copies of each partition.

A topic is a named stream of records, such as orders. Kafka splits it into partitions. Each partition is an ordered, append-only log stored on disk. New records are only added at the end.

Every record in a partition gets a number in sequence, called its offset. Consumers use offsets to track their position.

A broker is a Kafka server. Kafka spreads partitions across brokers, so the whole cluster shares a topic's load and storage.

Each partition has a replication factor: the number of copies it has. Each copy is a replica:

  • One replica is the leader. It handles writes and normally reads too.
  • The other replicas are followers. They copy the leader.
  • Followers that are fully caught up form the ISR (in-sync replica set).
  • If the leader's broker fails, a replica from the ISR becomes the new leader.

Partitions are the unit of parallelism and of ordering. Kafka guarantees order only inside one partition.

  • A topic is split into partitions, and each partition is an append-only log.
  • An offset is a record's position inside its partition.
  • Brokers host partitions, so the load spreads across the cluster.
  • Each partition has a leader and follower replicas, and a replica from the ISR takes over if the leader fails.
  • A partition is the unit of ordering and parallelism.