Dead Letter Queue

Jobs whose retries are exhausted should be sent to a dead-letter queue with enough context retained for safe recovery.

The dead-letter queue (DLQ) is a classic messaging-queue pattern: when message consumption fails after reaching the maximum retry count or the message cannot be routed, it is moved to a dedicated queue awaiting manual or programmatic handling, used to separate transient failures from permanent failures.

The dead-letter queue payload should contain at least: the original input, the error category, the stack trace or failure reason, the attempt count, the timestamp, and the source queue name; network and model failures should be retried, malformed input should go straight to the dead-letter queue, and the DLQ must be visible on the dashboard and prioritized for manual reprocessing.

A dead-letter queue is a fallback mechanism in messaging systems that routes messages that have failed processing multiple times to a separate queue rather than discarding them or retrying indefinitely.

Dead-letter queues provide a safety net for long-running tasks, ensuring that failed messages have a clear destination for post-hoc investigation.

The system's message passing is based on SQS, configured with dead-letter queues to handle failed messages.

Combined with a Dead Letter Queue (DLQ), messages that fail processing multiple times are isolated to facilitate troubleshooting without blocking the main workflow.

For uncertain data errors, the prudent approach is to quarantine suspicious data to a dead-letter queue rather than making hasty modifications.

Common dead-letter queue implementations include Apache Kafka's dedicated Topics, AWS SQS's Dead Letter Queue configuration, and RabbitMQ's dead-letter exchange.

A dead-letter queue (DLQ) is a special queue in a message queue system dedicated to storing messages that cannot be processed normally.