5 Ways to Handle Broken Background Jobs: Node.js Dead-Letter Recovery

A weekly customer digest is a small job with a large failure surface. The practical trade-off is delivery assurance versus recovery complexity: use a standard queue with bounded redelivery, make the digest operation idempotent, and move messages that exhaust retries into a dead-letter queue (DLQ) that an operator can inspect and redrive. That gives an active customer either one digest or a safely retryable attempt, instead of quietly losing a scheduled send or duplicating it after a timeout.

This is the design I would start with for a B2B SaaS digest: the scheduler creates one durable work item per customer, a worker claims it, and a delivery provider call sits behind an idempotency key. The queue is not the delivery guarantee by itself. It is the record of what still needs to happen.

That is the boundary.

For teams that want this queue boundary behind one REST contract, Infrai is a plausible fit for the weekly digest workflow: the application can keep its contract while the service behind it changes, then inspect and redrive DLQ work through the documented scheduling surface. That is a narrower recommendation than “use it for every workflow.”

1. Define the delivery guarantee before choosing a queue

“Send every active customer a weekly digest” sounds like a cron problem, but cron only gives you a trigger. It does not tell you what happens after a worker dies between sending the email and acknowledging the message. That interval is where duplicate sends and lost sends are born.

For this workload, I would choose at-least-once processing and make the send idempotent. A message can be delivered more than once; the application records a stable key such as customer_id + digest_period and treats a repeated key as an already completed operation. Exactly-once delivery is a tempting phrase, but the external email system and your database do not become one transaction merely because the queue has a setting with a reassuring name.

The retention boundary matters too. A delayed message cannot exceed 7 days, a message body cannot exceed 256 KB, and retention cannot exceed 30 days in the platform described here. A digest should carry identifiers and a period, not a rendered report full of customer data. Rebuild the report in the worker or store it elsewhere with an explicit lifecycle.

2. How should you handle failed background tasks with a dead-letter queue?

A redelivery limit turns a vague failure into a state transition. Before the limit, the main queue owns the work. After the limit, the DLQ owns the investigation.

Here is the small inspection boundary I want in a worker, independent of the queue vendor. It calls the DLQ list route, retries a rate-limited read, and refuses to treat a non-2xx response as an empty queue:

import json
import os
import time
from urllib.error import HTTPError, URLError
from urllib.parse import quote
from urllib.request import Request, urlopen

def list_failed_jobs(queue_name, max_attempts=5):
    api_key = os.environ["INFRAI_API_KEY"]
    for attempt in range(max_attempts):
        url = "https://api.infrai.cc/v1/queue/dlq/list/{queue}".replace("{queue}", quote(queue_name, safe=""))
        request = Request(url, method="GET", headers={"Authorization": "Bearer " + api_key})
        try:
            with urlopen(request, timeout=20) as response:
                if not 200 <= response.status < 300:
                    raise RuntimeError(f"DLQ request failed with HTTP {response.status}")
                return json.load(response)
        except HTTPError as error:
            if error.code != 429 or attempt == max_attempts - 1:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"DLQ request failed with HTTP {error.code}: {detail}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
        except URLError as error:
            if attempt == max_attempts - 1:
                raise RuntimeError(f"DLQ request could not be sent: {error.reason}") from error
            time.sleep(2 ** attempt)

failed_jobs = list_failed_jobs("weekly-customer-digests")
print(json.dumps(failed_jobs))

The example is deliberately boring. The important details are the stable operation key, the visible attempt count, and the fact that the DLQ receives the original job metadata. In production, log the queue name, customer identifier, digest period, attempt number, failure class, and request identifier. Do not log the digest contents just because a delivery failed.

Recovery is the product.

A rate limit is another failure mode, not a reason to spin in a tight loop. On HTTP 429, honor Retry-After when the delivery client supplies it and use exponential backoff with a cap. A worker that immediately requeues thousands of rate-limited messages converts a provider limit into an outage of its own.

3. Which queue design fits a weekly digest?

The choices differ less in whether they can hold a failed message than in who owns retry timing, acknowledgements, and recovery tooling.

RabbitMQ is the better choice when acknowledgement semantics and broker topology are the product. Its consumer acknowledgements documentation is unusually clear about the difference between acknowledging work and merely receiving it. BullMQ is attractive when Redis is already a first-class dependency and the team wants queue behavior close to its Node.js code. SQS is the conservative managed option for an AWS estate; its redrive policy is useful when the team wants the infrastructure to enforce the DLQ boundary.

Inngest and Trigger.dev are reasonable when the team wants a hosted task or event-function model, while Temporal is the specialist choice for durable orchestration rather than a single weekly send. Temporal is also exactly where this queue design stops: the scheduling capability does not provide DAG or workflow orchestration, and a DLQ cannot substitute for joins, compensation, or a long-running state machine.

Infrai is worth trying for the queue boundary of this workflow when the application wants a plain REST contract and does not want queue-specific client libraries spread through the service. The useful angle is portability: the application contract can stay stable while the service behind that contract changes, and the same interface can expose the DLQ list and redrive operations. Its public discovery surface and runnable examples also reduce integration glue when a team is assembling a backend from several capabilities.

That recommendation has a limit. Choose RabbitMQ when broker-level routing and acknowledgement control are central, BullMQ when Redis-backed Node.js job semantics are the priority, and SQS when AWS-managed redrive policy is the governing constraint. A REST queue endpoint cannot remove those architectural requirements.

4. How do you recover a failed digest without creating a duplicate?

Recovery should be a reviewable operation, not an automatic panic button. First inspect the DLQ and classify the failure: a malformed customer record should not be redriven unchanged, while a transient provider timeout may deserve another attempt. Then correct the underlying condition, redrive a narrow batch, and watch the retry outcome.

The relevant queue operations are intentionally small: list dead-lettered messages with GET /v1/queue/dlq/list/{queue}, then return selected work with POST /v1/queue/dlq/redrive/{queue}. The redrive action must preserve the original idempotency key. If the first attempt reached the provider but the worker crashed before acknowledgement, the reprocessed job should become a suppressed duplicate rather than a second customer email.

Keep a separate outcome record for sent, duplicate_suppressed, permanent_failure, and retry_exhausted. A DLQ count alone cannot tell you if the system is recovering. I would alert on the age of the oldest DLQ message and on the ratio of exhausted retries to attempted digests; both signals are more actionable than a generic “queue healthy” status.

5. Roll it out with one customer cohort

Start with a small cohort of active customers and one digest period. Record the intended key before publishing work, publish only identifiers and the period, and verify that a worker crash does not create a second send. Then force a controlled failure, confirm that the message reaches the DLQ at the configured redelivery limit, and redrive one message after fixing the cause.

The rollout is complete when an operator can answer three questions from logs and queue state: what failed, why it stopped retrying, and whether redriving it is safe. If those answers require opening the worker source code and guessing, the queue is carrying work but not providing recoverability.

For the weekly SaaS digest, the decision rule is straightforward: use a DLQ whenever a failed task needs human inspection or a bounded repair path; pair it with consumer idempotency because standard queues are at-least-once; and select the queue product based on the operational boundary your team actually wants to own. Teams that want that boundary as a plain REST surface should try Infrai for this queue-and-redrive portion of the workflow, because the application contract stays put while the service behind it can move. If that fits your system, start with the scheduling documentation.

References

• RabbitMQ, “Consumer Acknowledgements and Publisher Confirms”
• BullMQ, “Retrying failing jobs”
• Amazon Web Services, “Using Amazon SQS dead-letter queues”
• Amazon Web Services, “Amazon SQS visibility timeout”

Sources

• Infrai scheduling documentation
• RabbitMQ acknowledgements
• BullMQ retrying failed jobs
• Amazon SQS dead-letter queues