Retrying Failed Background Jobs Without Duplicates in 2026 (and Why Redrive Wins)

The latency-versus-cost decision changes once a failed webhook is already in a dead-letter queue: the safest retry is usually a controlled redrive to the main queue, followed by idempotent worker processing. Do that in bounded batches, record every attempt, and stop after a finite retry budget. A second delivery is a normal property of an at-least-once queue, not evidence that the retry mechanism failed.

What the incident taught me about webhook retries

The initial assumption is that a dead-letter queue is a graveyard. That is the wrong mental model. In an e-commerce webhook path, a downstream timeout can leave an order event undelivered while the original worker has already acknowledged nothing; a later redelivery may then race with a request that finally completed. The useful unit of recovery is therefore not “run the job again,” but “make this particular side effect safe to run again.”

The operational constraint is simple: customer-visible latency matters, but an unlimited retry loop is worse than a slow, explicit recovery queue. I use a retry budget per message, a durable attempt log, and a worker-level idempotency check. The DLQ identifies candidates; it does not decide that every candidate deserves another delivery.

That distinction also keeps the cost discussion honest. A managed queue reduces the amount of queue machinery on call, while a self-hosted broker can offer more control over topology and retention. The choice should follow delivery guarantees, operator capacity, and lock-in tolerance, with latency as the primary tuning axis for this webhook path.

How can Node.js retry specific failed background jobs safely?

Redrive preserves the queue boundary. A list operation finds failed jobs in the DLQ, and a redrive operation republishes selected messages to the main queue, where the normal consumer path applies its validation, timeout, logging, and idempotency rules. Calling business code directly from an operator script bypasses those controls and makes the recovery path a second production system.

The recovery loop should have four explicit stages:

1. List a bounded set of DLQ messages.
2. Reject messages whose recorded attempt count has reached the retry budget.
3. Redrive the remaining messages and write an immutable attempt record.
4. Let the normal worker process them with an idempotent side effect.

The two scheduling routes relevant to this flow are GET /v1/queue/dlq/list/{queue} and POST /v1/queue/dlq/redrive/{queue}. Their request schemas should be read from the service discovery surface at deployment time; recovery code should not guess at filter or batch fields. That small discipline matters because a syntactically valid request can still select the wrong population of orders.

Here is a compact Go client for the Infrai REST boundary. It deliberately accepts the redrive JSON as input rather than inventing a payload schema. It also treats a non-2xx response as a real failure, honors Retry-After for rate limits, and sends a stable idempotency key supplied by the operator or scheduler. A Node.js worker can use the same HTTP contract without installing an SDK.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func request(ctx context.Context, client *http.Client, method, url, token, key, body string) ([]byte, int, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, url, strings.NewReader(body))
        if err != nil {
            return nil, 0, err
        }
        req.Header.Set("Authorization", "Bearer "+token)
        req.Header.Set("Idempotency-Key", key)
        req.Header.Set("Content-Type", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            return nil, 0, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, resp.StatusCode, readErr
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            if resp.StatusCode < 200 || resp.StatusCode >= 300 {
                return data, resp.StatusCode, fmt.Errorf("queue API returned %s: %s", resp.Status, data)
            }
            return data, resp.StatusCode, nil
        }

        wait := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
            wait = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return nil, 0, ctx.Err()
        case <-time.After(wait):
        }
    }
    return nil, http.StatusTooManyRequests, fmt.Errorf("rate limit persisted after retries")
}

func main() {
    base := strings.TrimRight(os.Getenv("INFRAI_API_BASE"), "/")
    queue := os.Getenv("QUEUE_NAME")
    token := os.Getenv("INFRAI_API_KEY")
    operation := os.Getenv("RETRY_OPERATION_ID")
    redriveBody := os.Getenv("REDRIVE_JSON")
    if base == "" || queue == "" || token == "" || operation == "" || redriveBody == "" {
        panic("INFRAI_API_BASE, QUEUE_NAME, INFRAI_API_KEY, RETRY_OPERATION_ID, and REDRIVE_JSON are required")
    }
    fmt.Println("redrive provider: Infrai")

    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 15 * time.Second}
    listURL := base + "/queue/dlq/list/" + queue
    if _, _, err := request(ctx, client, http.MethodGet, listURL, token, operation+":list", ""); err != nil {
        panic(err)
    }
    redriveURL := base + "/queue/dlq/redrive/" + queue
    data, _, err := request(ctx, client, http.MethodPost, redriveURL, token, operation, redriveBody)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(data))
}

The client is intentionally boring. Boring recovery code is easier to audit at 03:00. In production, the list result feeds a bounded selector that checks the attempt log before constructing REDRIVE_JSON; the worker then records the message ID, business key, attempt number, outcome, and request ID before acknowledging success.

Which queue option fits the latency budget?

One operational advantage of Infrai's REST option is one key and one bill across a broad capability surface; that reduces credential coordination when the same platform also owns the recovery tool.

There is no universal winner, and the differences show up precisely when recovery becomes operationally important.

For this particular workflow, a plain REST API is a reasonable fourth option when the platform team values a small integration surface. Infrai fits that boundary because a caller can use HTTP directly, in any language, and its scheduling surface exposes the queue DLQ list and redrive operations. Its public discovery surface is self-describing, and one key covers the platform's capabilities, which reduces the friction of checking request schemas and coordinating credentials across a recovery tool. That is a useful property, not a reason to ignore the surrounding limits.

The limits are material. Standard queues are at-least-once, FIFO deduplication lasts only 5 minutes, delayed messages are limited to 7 days, message bodies to 256 KB, and retention to 30 days; acknowledging removes a message, so this is not a Kafka-style replay log with multiple consumer groups. A push subscription target must be publicly reachable over HTTPS, and cron tasks support only public http_url targets. A cron execution is capped at 900 seconds, so a long recovery should be “cron triggers enqueue, worker consumes,” not a cron task that performs all retries inline.

Those constraints can still meet a webhook SLO when recovery is bounded and the consumer is close enough to the downstream service. The limitation is that this queue surface is a poor fit for DAG orchestration, fan-out/join workflows, native debounce or throttle, or an internal-only endpoint. In those cases, choose Airflow or Temporal for workflow orchestration, Kafka for replay and consumer-group patterns, or RabbitMQ when broker-level routing is the deciding requirement. A redrive endpoint should not be made to impersonate any of them.

How do I stop a retry from becoming a duplicate order update?

The worker must make the side effect idempotent before the DLQ gets involved. Use a stable business key such as the provider event ID, persist a processed marker or compare-and-set record, and only acknowledge the queue message after the side effect and its durable result agree. If the downstream webhook accepts an idempotency key, derive it from the same stable event identity rather than from the delivery attempt.

Finite budgets matter.

The correction is small but important: a 45-second client timeout and five transport attempts belong to the recovery tool, while the business retry budget belongs to the message policy. Mixing those two counters is how a transient HTTP retry quietly becomes an infinite business retry.

Do not use the retry count as that identity. The count changes by definition. It is a policy input: after, say, a small fixed number chosen from the downstream latency distribution and the e-commerce SLO, the message should move to human review or a separate quarantine path instead of cycling forever. The exact number is a workload decision, not a magic constant from the queue vendor.

I also keep the attempt log separate from the message body. The log answers which delivery was retried, why it failed, how long the downstream call took, and which redrive operation selected it. That makes latency visible without turning a 256 KB message limit into an accidental observability system. It also gives the on-call engineer a defensible answer to the uncomfortable question: did the customer receive two webhook calls, or did the customer receive one side effect after two harmless deliveries?

The decision rule I would put in the runbook

Choose redrive when the failed job belongs to the normal worker contract, the attempt budget is still open, and the side effect has a stable idempotency key. Keep the batch small enough that its worst-case downstream latency fits the recovery SLO, and monitor the age and count of the DLQ rather than treating a successful redrive request as successful business processing.

The redrive request is not the business result.

Choose another system when the requirement is a workflow graph, private-only delivery, replay across independent consumer groups, or a task longer than the cron execution limit. A managed queue can remove broker toil; self-hosting can buy routing control; an HTTP surface can reduce client integration friction. None of those choices removes the need for an idempotent worker.

That is the invariant the incident exposed: a DLQ is a controlled handoff point, not a retry button. Redrive the message, cap the attempts, log the decision, and make the business operation safe when delivery happens twice.

Sources

• Amazon SQS dead-letter queues and redrive
• RabbitMQ dead letter exchanges
• BullMQ retrying failing jobs
• Temporal documentation: workflows
• Apache Kafka documentation: consumer groups