Explain the trade-off between message ordering and parallelism. How do you get both where it matters?

You cannot have total order and full parallelism together. But you rarely need total order, only order per entity. So split messages by entity key and process the groups in parallel.

Why they conflict:

• Strict ordering means processing one message at a time. Message 2 cannot start until message 1 is done.
• Parallelism means processing several messages at once. They can then finish in any order.

Per-entity order means order for the events of one order, one account or one user. To get it, partition by that entity's key. In Kafka, records with the same key go to the same partition, and a consumer reads one partition in order. Different partitions are processed in parallel. The number of partitions is the limit on parallelism.

You can go further inside one consumer. Process different keys at the same time, but keep each key in sequence. For example, keep a small queue per key.

Watch out: batches and async handlers that finish out of order can quietly break per-key ordering.