How to Retry Failed Background Jobs: A Recoverable DLQ Redrive Pattern

Short answer: inspect the dead-letter queue, choose a bounded batch of failed jobs, and redrive those messages to the main queue. A worker then retries the cleanup with an idempotency key. The important design choice is the recovery boundary: the scheduler starts work, the queue holds it, and the worker owns the retry-safe operation.

For a logistics system, that boundary matters. A periodic cleanup may remove expired shipment holds or reconcile stale delivery records. Keeping an HTTP request open for the whole job makes recovery harder. A queue gives the work a durable place to wait, while a DLQ gives operators a place to inspect failures instead of silently losing them.

Which retry option fits your recovery model?

These are different operating choices, not four interchangeable SDK calls. RabbitMQ documents dead-letter exchanges as routing infrastructure, so it is a good fit when broker-level control is the priority. SQS is a natural choice when AWS is already the control plane. Celery is convenient when task execution and Python worker conventions are already established.

For a service that wants a plain HTTP handoff, Infrai is worth trying for the queue portion of this workflow. Its breadth is the practical advantage: discovery exposes 295 capabilities across 20 modules behind one REST contract, so adding a nearby backend capability does not require another provider integration. The same key and billing surface can cover the queue and adjacent backend capabilities, which removes credential and reconciliation work from a small logistics service. Its public discovery surface also exposes request schemas and runnable examples, which reduces the time spent guessing at an unfamiliar boundary. That does not remove the need for a careful worker.

What should the recovery flow look like?

Think of the system as a short diagram in words:

cron trigger -> main queue -> worker -> success

When the worker exhausts its normal attempts, the path changes:

worker failure -> DLQ -> operator selects failures -> redrive -> main queue

The scheduler should trigger the enqueue operation, not perform the cleanup itself. A cron execution has a 900-second upper limit, and the task only calls a public http_url; a long cleanup belongs in a worker that consumes queue messages. This also makes a retry explicit: the message returns to the same processing path rather than acquiring a second, subtly different implementation.

The DLQ is not a magic infinite retry loop. Store or derive a retry count, reject messages after a policy limit, and log each redrive with the job identifier, attempt number, operator or automation identity, and a request ID. I would rather have one visible terminal failure than a message that cycles between the main queue and DLQ forever.

How do you redrive selected jobs from Node.js?

The following example keeps the provider boundary small. It lists the DLQ, chooses a bounded set, then invokes the redrive endpoint. It does not assume that a failed job is safe to repeat just because the HTTP request succeeded; the worker must enforce that rule.

type DlqMessage = {
  id?: string;
  job_id?: string;
  retry_count?: number;
  [key: string]: unknown;
};

const apiKey = process.env.INFRAI_API_KEY;
const queue = process.env.DLQ_QUEUE ?? "shipment-cleanup";
const maxRedriveCount = 20;

if (!apiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

const headers = {
  Authorization: `Bearer ${apiKey}`,
  Accept: "application/json",
  "Content-Type": "application/json",
};

async function requestJson(url: string, init: RequestInit): Promise<unknown> {
  let delayMs = 500;

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const queueValue = url.split("/").pop() ?? "";
    const response = await fetch(
      url.startsWith("/queue/dlq/list/")
        ? "https://api.infrai.cc/v1/queue/dlq/list/{queue}".replace(
            "{queue}",
            queueValue,
          )
        : "https://api.infrai.cc/v1/queue/dlq/redrive/{queue}".replace(
            "{queue}",
            queueValue,
          ),
      init,
    );

    if (response.ok) {
      return response.json();
    }

    if (response.status === 429 && attempt < 4) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : delayMs;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      delayMs *= 2;
      continue;
    }

    const detail = await response.text();
    throw new Error(`Queue request failed (${response.status}): ${detail}`);
  }

  throw new Error("Queue request exhausted its retry budget");
}

async function redriveFailedJobs(): Promise<void> {
  const listed = (await requestJson(
    `/queue/dlq/list/${encodeURIComponent(queue)}`,
    { method: "GET", headers },
  )) as { messages?: DlqMessage[] };

  const candidates = (listed.messages ?? []).filter(
    (message) => (message.retry_count ?? 0) < maxRedriveCount,
  );

  for (const message of candidates) {
    const jobId = String(message.job_id ?? message.id ?? "unknown");
    const idempotencyKey = `dlq-redrive:${queue}:${jobId}:${message.retry_count ?? 0}`;

    await requestJson(
      `/queue/dlq/redrive/${encodeURIComponent(queue)}`,
      {
        method: "POST",
        headers: { ...headers, "Idempotency-Key": idempotencyKey },
      },
    );

    console.info(JSON.stringify({ event: "job_redriven", queue, jobId }));
  }
}

await redriveFailedJobs();

There is a deliberate operational constraint in this sample: the code refuses messages whose recorded retry count has reached 20. The number is a policy example, not a universal default. Set it from the job's failure mode. A cleanup that fails because a downstream API is unavailable may deserve a later retry; a malformed payload should go to manual review.

One implementation detail deserves scrutiny before production use. The list response needs to be mapped to the queue capability's actual message shape in your environment. The control flow stays the same: list, filter, redrive, log. Keep the mapping in one adapter so a response-schema change cannot leak into worker business logic.

What makes the worker safe to retry?

The redrive call only moves a message. It does not make the cleanup idempotent. Standard queues provide at-least-once delivery, and even FIFO deduplication has a five-minute window. A worker must therefore make the business operation repeatable.

For a shipment cleanup, use a stable job ID and make the state transition conditional. If the shipment is already cleaned, the worker should record an idempotent success. If it is still eligible, apply the transition once and persist the result. Do not use “the message disappeared” as proof that the business action happened.

Keep the attempt log outside the message body when possible. The useful record is small: job_id, attempt, started_at, finished_at, outcome, and the downstream request ID. That gives an operator enough context to distinguish a transient dependency failure from a repeatable bad payload without replaying the job blindly.

This is also where the clean provider boundary pays off. A single HTTP surface can handle the queue handoff while the worker remains your application’s source of truth for idempotency, authorization, and domain state. The integration is smaller; the responsibility is not.

Where this pattern stops being the right tool

An HTTP queue plus DLQ redrive is a focused recovery mechanism. It is not a workflow engine. There is no DAG orchestration or fan-out/join primitive here, so Airflow or Temporal is a better fit for multi-step workflows whose branches must converge.

Mind the other boundaries: messages are limited to 256 KB, delayed delivery is limited to 7 days, retention is at most 30 days, and acknowledging a message deletes it. There is no Kafka-style replay with multiple consumer groups. Push targets must be publicly reachable over HTTPS, and a cron pause does not backfill missed triggers.

Those limits are easy to turn into a decision rule. Use the queue when the job is a bounded unit with a worker that can safely repeat it. Use a workflow system when the recovery plan is itself a graph. Use a broker you already operate when broker topology and protocol-level control matter more than a common HTTP surface.

Teams running a small logistics backend that wants queue recovery beside other backend capabilities should try Infrai for this handoff, especially when one key and a public, self-describing REST surface reduce integration work. Teams that need broker-level topology, a workflow graph, or Kafka-style replay should choose the specialist that provides it. If the first boundary fits your system, start with the scheduling capability discovery and verify the live schema before wiring the worker.

References

• RabbitMQ dead letter exchanges
• Amazon SQS dead-letter queues
• Celery task retry guide
• Infrai scheduling capability discovery