CodePark/ questions / BullMQ & Node job queues
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › BullMQ & Node job queues

What are BullMQ's main building blocks, and what states does a job move through?

BullMQ is a job queue for Node that stores everything in Redis. It has three main building blocks:

  • A Queue adds jobs: queue.add(name, data, opts).
  • A Worker takes jobs from one queue and runs a processor function on each job.
  • QueueEvents listens to events for the whole queue, such as completed, failed and progress. It works from any process, because it reads them from Redis streams.

Jobs, their data and the queue state all live in Redis. Because of this, the producers (the code that adds jobs) and the workers can run in different processes or on different servers.

A job moves through these states:

  • It starts in waiting. If it has a delay, it starts in delayed. If it has a priority, it starts in prioritized.
  • A worker moves it to active while it runs.
  • It ends in completed, with a return value, or in failed, once it has used all its attempts.
  • A retry with backoff (a wait before the next try) moves a failed attempt back to delayed.
  • In a flow (a parent job with child jobs), the parent waits in waiting-children until its children finish.
  • A Queue adds jobs, a Worker processes them, and QueueEvents listens to their events.
  • All state lives in Redis, so producers and workers are independent of each other.
  • The main states are waiting, delayed or prioritized, then active, then completed or failed.
  • Retries go back through delayed, and parent jobs in flows wait in waiting-children.