CodePark/ questions / Data types & commands
START PRACTICINGPRACTICE
Backend question · Redis & caching › Data types & commands

How can you build a simple job queue on a Redis list, and how do you avoid losing a job if the worker crashes?

Producers add jobs with LPUSH, and workers take them from the other end with BRPOP. BRPOP blocks (waits) until an item arrives, so workers don't need to poll. Each item goes to exactly one worker.

The weakness: once a worker pops a job, the job exists only in that worker's memory. If the worker crashes, the job is gone.

The reliable-queue pattern fixes this:

  • The worker uses LMOVE, or its blocking form BLMOVE, to move the job from the queue into its own processing list. The move is atomic, so the job is always in one of the two lists.
  • When the job finishes, the worker removes it from the processing list with LREM.
  • A separate cleanup process (a janitor) checks the processing lists for stuck jobs and pushes them back onto the queue.

BLMOVE replaces the older BRPOPLPUSH command.

Do you need acknowledgements, retries and multiple consumer groups built in? Then Redis Streams are usually a better choice than lists you build yourself.

  • Use LPUSH to add jobs and BRPOP to wait for and take jobs.
  • A plain pop loses the job if the worker crashes.
  • Use LMOVE or BLMOVE to move the job into a processing list, and LREM when it is done.
  • A recovery process puts stuck jobs back on the queue.
  • Redis Streams give you acknowledgements and consumer groups built in.