CodePark/ questions / BullMQ & Node job queues
START PRACTICINGPRACTICE
Backend question · Message queues & streaming › BullMQ & Node job queues

What is a stalled job in BullMQ, what causes it, and what are the consequences?

A stalled job is an active job whose lock in Redis has expired, so BullMQ thinks its worker is gone.

How it works: when a worker takes a job, it gets a lock in Redis for lockDuration (30 seconds by default). The worker renews this lock again and again while the job runs. BullMQ regularly checks for active jobs whose lock has expired. When it finds one, the job is stalled. Then one of two things happens:

  • The job moves back to waiting, and another worker picks it up.
  • The job fails, if it has stalled more than maxStalledCount times (1 by default).

Two causes:

  • The worker process died. For example, it crashed, the system killed it for using too much memory (OOM kill), or a deploy stopped it without a graceful shutdown. Stall detection exists for this case.
  • The worker is still alive, but CPU-heavy synchronous code blocked its event loop. So it could not renew the lock in time.

Consequence: in the second case, the job runs twice at the same time. The old worker is still running it while a new worker starts it again. So side effects, such as sending an email, can happen twice.

Fixes:

  • Keep the event loop free. Use sandboxed processors for CPU work.
  • Close workers gracefully when the app shuts down.
  • Make processors idempotent, which means running a job twice has the same effect as running it once.
  • A worker holds a lock for each active job and renews it, with a lockDuration of 30 s by default.
  • When the lock expires, the job is stalled and goes back to waiting, or it fails after maxStalledCount stalls.
  • Stalls come from a crashed worker, or from a blocked event loop that cannot renew the lock.
  • If the worker is blocked but still alive, the job can run twice.
  • Fix it with sandboxed processors, graceful shutdown and idempotent processors.