BullMQ
A Redis-based message queue and job scheduling library for Node.js, supporting delayed jobs, retries, deduplication (jobId), and other features. It is commonly used for background tasks and workflow orchestration.
Timeline (last 90 days)
The criterion for choosing queue tooling is that the team can still operate it correctly at a 2 a.m. outage: BullMQ suits Node.js workers, Temporal suits multi-step orchestration with long waits and approvals, AWS SQS suits AWS-native managed queues, and Cloudflare Queues suits edge-friendly batch-processing architectures.
Having cron directly execute the full Agent logic (such as */15 * * * * node run-agent.js) is a bad pattern; the standard pattern is for cron only to enqueue (such as node enqueue-agent-job.js, writing to BullMQ, SQS, or Cloudflare Queues), with a separate worker executing the Agent steps.
BullMQ's exponential backoff formula is 2 ^ (attempts - 1) * delay; with a base delay of 3000ms, the 7th attempt waits about 192 seconds (about 3.2 minutes).
BullMQ's jobId deduplication is only effective while the old job still exists; if completed or failed jobs are aggressively cleaned up, the deduplication window disappears with them.
These long-running process options (such as Celery, BullMQ, Kafka consumers, Temporal workers) were previously difficult to deploy directly on pure Serverless platforms, and often required a separate container hosting service.
Options in the same category as Celery also include BullMQ (Redis-based) in the Node.js ecosystem, Python's RQ, cross-language Kafka consumer groups, and Temporal workflow workers.
Parallel processing in engineering implementations typically relies on cloud function or task queue architectures, with common implementations including AWS Lambda, Azure Functions, Celery, and BullMQ.
Because LLM inference latency is typically on the order of several seconds, synchronous HTTP requests are prone to triggering gateway timeouts, and production deployments often need to introduce task queues (such as Celery or BullMQ) or WebSocket/SSE to maintain persistent connections.
Handling high concurrency in agents typically requires introducing asynchronous task queues (such as Celery or BullMQ), performing streaming decomposition of long tasks, setting timeout and circuit-breaker mechanisms for tool calls, and distinguishing between fast-response channels and background execution channels.
The default implementation of most queue libraries (such as Celery, Sidekiq, and BullMQ) only serializes business parameters, does not automatically carry initiator metadata, and does not distinguish between the current user and the actual operator.
All Facts (10)
The criterion for choosing queue tooling is that the team can still operate it correctly at a 2 a.m. outage: BullMQ suits Node.js workers, Temporal suits multi-step orchestration with long waits and approvals, AWS SQS suits AWS-native managed queues, and Cloudflare Queues suits edge-friendly batch-processing architectures.
50%UnverifiedHaving cron directly execute the full Agent logic (such as */15 * * * * node run-agent.js) is a bad pattern; the standard pattern is for cron only to enqueue (such as node enqueue-agent-job.js, writing to BullMQ, SQS, or Cloudflare Queues), with a separate worker executing the Agent steps.
50%UnverifiedBullMQ's exponential backoff formula is 2 ^ (attempts - 1) * delay; with a base delay of 3000ms, the 7th attempt waits about 192 seconds (about 3.2 minutes).
50%UnverifiedBullMQ's jobId deduplication is only effective while the old job still exists; if completed or failed jobs are aggressively cleaned up, the deduplication window disappears with them.
50%UnverifiedThese long-running process options (such as Celery, BullMQ, Kafka consumers, Temporal workers) were previously difficult to deploy directly on pure Serverless platforms, and often required a separate container hosting service.
50%UnverifiedOptions in the same category as Celery also include BullMQ (Redis-based) in the Node.js ecosystem, Python's RQ, cross-language Kafka consumer groups, and Temporal workflow workers.
50%UnverifiedParallel processing in engineering implementations typically relies on cloud function or task queue architectures, with common implementations including AWS Lambda, Azure Functions, Celery, and BullMQ.
50%UnverifiedBecause LLM inference latency is typically on the order of several seconds, synchronous HTTP requests are prone to triggering gateway timeouts, and production deployments often need to introduce task queues (such as Celery or BullMQ) or WebSocket/SSE to maintain persistent connections.
50%UnverifiedHandling high concurrency in agents typically requires introducing asynchronous task queues (such as Celery or BullMQ), performing streaming decomposition of long tasks, setting timeout and circuit-breaker mechanisms for tool calls, and distinguishing between fast-response channels and background execution channels.
50%UnverifiedThe default implementation of most queue libraries (such as Celery, Sidekiq, and BullMQ) only serializes business parameters, does not automatically carry initiator metadata, and does not distinguish between the current user and the actual operator.
50%