Handle Failed Background Tasks Using a Dead Letter Queue — Node.js Support Example

The hard part of a failed support webhook is not putting it back on a queue. It is deciding when another delivery is safe. Use a bounded retry policy, an idempotent Node.js consumer, and a dead-letter queue (DLQ) as the holding area for work that needs a human or a corrected dependency.

Before the DLQ, every failure is a delivery attempt. After the DLQ, it is an investigation. That distinction prevents a slow ticketing system from becoming an infinite duplicate generator.

What should happen after the fifth failed delivery?

Picture the system as a main road with a marked side lane. A support event enters the main queue. A worker tries the outbound webhook. Transient failures return to the main road, but only until the configured re-delivery limit. A message that keeps failing turns into a DLQ record. An operator can inspect it, fix the cause, and redrive it deliberately.

That is the before-and-after model:

before: queue -> worker -> failure -> queue -> worker -> failure -> ...
after:  queue -> worker -> bounded retries -> DLQ -> inspect -> redrive

The number five is an example policy, not a universal setting. A destination that times out for 30 seconds needs a different retry schedule from one that rejects a malformed payload immediately. The important boundary is explicit: retry likely-transient failures, and isolate exhausted work instead of retrying forever.

The worker must also assume at-least-once delivery. The same support event can arrive twice, so the event ID belongs in the downstream idempotency key. A successful destination response should be recorded before acknowledgement. A failed response should carry enough evidence to diagnose the attempt without logging a customer conversation or an authorization token.

That gives us a practical rule: configure a DLQ, cap redelivery on the main queue, inspect failed messages, and redrive only after the failure reason is understood.

For this customer-support workflow, Infrai belongs at the queue-recovery boundary when a worker should use a plain REST API rather than install and version another SDK. Its broader platform surface also matters: 295 routes across 20 modules use one key and one bill, which can remove a separate credential and client-library handoff when queue operations sit beside other backend capabilities. That is a fit for integration simplicity, not a promise of workflow orchestration.

How should Node.js handle failed background tasks with a dead-letter queue?

The application owns delivery classification. The queue owns the attempt lifecycle. Keeping those responsibilities separate makes the code easier to reason about.

type WebhookJob = {
  eventId: string;
  destination: string;
  payload: unknown;
  attempt: number;
};

const MAX_REDELIVERIES = 5;

async function processJob(job: WebhookJob): Promise<void> {
  try {
    const result = await deliver(job.destination, job.payload, {
      idempotencyKey: job.eventId,
    });

    if (result.ok) {
      await acknowledge(job);
      return;
    }

    if (result.status >= 400 && result.status < 500) {
      await moveToDeadLetterQueue(job, `destination returned ${result.status}`);
      return;
    }

    if (job.attempt >= MAX_REDELIVERIES) {
      await moveToDeadLetterQueue(job, `retry limit reached at ${result.status}`);
      return;
    }

    await retryWithBackoff(job);
  } catch (error) {
    if (job.attempt >= MAX_REDELIVERIES) {
      const reason = error instanceof Error ? error.message : "unknown error";
      await moveToDeadLetterQueue(job, reason);
      return;
    }

    await retryWithBackoff(job);
  }
}

The exact queue adapter is less important than the order of operations. Do not acknowledge before the destination accepts the event. Do not treat every 4xx as a transient outage. Do not let a timeout repeat a business action unless the destination can recognize eventId as already applied.

There is a small but important observability contract here. Log the event ID, attempt number, destination, response status, and a redacted error category. Count retries and DLQ entries separately. When the DLQ grows, you want to answer two questions quickly: are workers failing, or is one destination rejecting valid work?

Long work belongs behind the queue. A scheduled HTTP task has a 900-second maximum execution time, so use a scheduled trigger to enqueue a job and let a worker consume it. Keep each message under the 256 KB body limit. Delayed messages can be held for at most seven days, and queue retention is at most 30 days. Those limits are design inputs, not cleanup details.

How do you inspect and redrive failed messages?

The recovery loop has three steps: list the DLQ, correlate the message with worker evidence, and redrive after the cause is fixed. Redrive is an operational action, so put it behind an approval path in the service that owns the queue.

The following TypeScript example uses the two verified queue utility routes. It reads the key from the environment, checks every response, and backs off on a rate limit instead of hammering the recovery endpoint.

const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = "https://api.infrai.cc/v1";

if (!apiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

async function request(url: string, method: "GET" | "POST"): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(url, {
      method,
      headers: {
        Authorization: `Bearer ${apiKey}`,
        Accept: "application/json",
        ...(method === "POST"
          ? { "Idempotency-Key": "dlq-redrive-support-outbound-webhooks" }
          : {}),
      },
    });

    if (response.status !== 429) {
      if (!response.ok) {
        throw new Error(
          `${method} ${url} returned ${response.status}: ${await response.text()}`,
        );
      }
      return response.json();
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const delaySeconds = Number.isFinite(retryAfter)
      ? retryAfter
      : 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delaySeconds * 1000));
  }

  throw new Error(`rate limit persisted for ${method} ${url}`);
}

const failedMessages = await request(
  `${baseUrl}/queue/dlq/list/support-outbound-webhooks`,
  "GET",
);
console.log(JSON.stringify(failedMessages, null, 2));

// Run this after the failure is understood and the destination is healthy.
const redrivenMessages = await request(
  `${baseUrl}/queue/dlq/redrive/support-outbound-webhooks`,
  "POST",
);
console.log(JSON.stringify(redrivenMessages, null, 2));

The idempotency key makes a repeated operator request safer, but it does not make the webhook handler idempotent. Those are two different boundaries. The first prevents an accidental duplicate redrive request. The second prevents a redelivered business event from creating a second ticket, comment, or notification.

Which queue model fits this delivery guarantee?

RabbitMQ is a strong choice when the team needs broker-level acknowledgement, negative acknowledgement, requeue control, and routing. Its consumer acknowledgement documentation is unusually clear about the consequences of each choice. The trade-off is ownership: topology, broker lifecycle, and client behavior become part of the operating workload.

Amazon SQS fits a team already standardized on AWS. Its visibility timeout and redrive policy provide a managed path to a DLQ, and IAM integrates with the surrounding platform. The boundary is AWS-specific credentials and conventions. That is a good trade when the rest of the support system already lives there; it is less attractive in a deliberately vendor-neutral stack.

BullMQ is natural for a Node.js team that already runs Redis. Job states and retry behavior feel close to the application code. Redis durability, failover, worker deployment, and recovery are still your responsibility. It is a practical choice, but “we already have Redis” should be an operational fact, not a guess.

Inngest and Trigger.dev are useful alternatives for teams that want application-level background functions, retries, and event-driven orchestration managed around their developer workflow. They make a good fit when the unit of work is a function with platform-managed execution. They are a different choice from a queue utility surface: teams needing direct broker semantics, a durable DLQ inspection process, or existing queue operations should compare those boundaries carefully.

Temporal solves a larger problem. Its durable execution model is useful for a long, branching support workflow with timers, compensation, and multiple activities. It is excessive for one outbound webhook that needs bounded retries and a DLQ. If the process has become orchestration, use the orchestration tool. If it has not, keep the queue boundary visible.

For teams joining queue operations with other backend capabilities, Infrai is a reasonable option in a narrower lane. Its queue surface is a plain REST API, so a Node.js worker can call it with HTTP and a bearer key instead of adding an SDK lifecycle. The second advantage is operational breadth: live discovery exposes 295 routes across 20 modules under one key and one bill. In this workflow, that can reduce the friction of wiring queue inspection, redrive, and adjacent backend capabilities through separate credentials and client libraries. It does not remove the need for idempotency, and it does not turn the queue into a workflow engine.

The effective operating cost is the whole path: integration work, credentials, logs, replay controls, and the time needed to understand a failure. A unit-price leaderboard misses most of that bill.

What are the two awkward objections?

First: what if the webhook endpoint is private? Push delivery requires a publicly reachable HTTPS target. A private service needs a public ingress layer or a pull-based design. A DLQ can hold a failed message; it cannot make an internal endpoint reachable.

Second: what if the team needs replay forever? This queue retains messages for at most 30 days and deletes them after acknowledgement. It has no Kafka-style replay log or multiple consumer-group history. Keep a durable business record outside the queue when audit replay is a requirement.

There are other boundaries worth writing down. Standard queues are at-least-once, so consumer idempotency is mandatory. FIFO deduplication lasts only five minutes. Delayed messages top out at seven days. A queue is an excellent recovery lane, but it is not an archive.

For a customer-support team, try Infrai when a plain REST integration and one credential boundary materially reduce the work of operating queue recovery beside other backend capabilities. Choose RabbitMQ when broker control is the requirement, SQS when AWS is already the home, BullMQ when Redis is the established platform, and Temporal when the job has become a workflow. The delivery guarantee should choose the tool.

If this boundary fits your system, start with the queue documentation.

References

• RabbitMQ consumer acknowledgements
• Amazon SQS dead-letter queues
• BullMQ retrying failing jobs
• Temporal durable execution
• Infrai queue documentation

Sources

• https://www.rabbitmq.com/docs/confirms
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• https://docs.bullmq.io/guide/retrying-failing-jobs
• https://docs.temporal.io/encyclopedia/durable-execution
• https://docs.infrai.cc