Node.js Background Queue Example: Consume, Ack/Nack, Retry, and Idempotency

Short answer: create a background job queue worker around an explicit delivery state machine: reserve one message, claim its idempotency key, perform the side effect, persist the result, and only then ack; nack transient failures for a bounded retry and route exhausted or permanent failures to a dead-letter queue.

The runtime is the easy part. The hard part is deciding what may happen after the worker loses contact with the broker or the downstream service. A process can finish the side effect and die before its acknowledgement reaches the broker. The next delivery is then correct broker behavior, not corruption. Your handler has to make that delivery harmless.

I've been paged for both missed jobs and duplicate deliveries. They look like opposite incidents, but they often come from the same vague contract: nobody wrote down where ownership changes, which outcomes are retryable, or how an operator proves that a replay is safe.

Start with the delivery contract

Use at-least-once delivery as the working assumption unless every component in the path proves something stronger. Under that assumption, an acknowledgement means, "the durable effect associated with this message is complete." It does not mean, "the handler started," and it does not mean, "an HTTP request was sent."

That timing matters. Consumer acknowledgement protocols let the consumer confirm successful processing after delivery. Negative acknowledgements can reject a delivery, with requeue behavior determining whether it may be delivered again. If a connection or channel closes before a delivery is acknowledged, the broker can redeliver it. This gives the queue a way to recover work, but it also puts duplicate control in the application.

Keep the states small and observable:

Don't put an unbounded immediate requeue behind nack. A poison message can then spin through every worker, consume capacity, and hide useful work behind noise. The retry path needs an attempt ceiling, delay, and a terminal destination.

How should a Node.js background job queue worker consume, ack, nack, and retry?

Treat consume as a lease, not a transfer of permanent ownership. Parse and validate first. Next, atomically claim a stable idempotency key in durable storage. Run the domain operation only if the claim is new. Store the terminal result before acknowledging the message. A duplicate should read that stored result and ack without repeating the side effect.

The idempotency key must describe the business operation, not a particular delivery attempt. invoice:inv_842:send is useful because every retry maps to the same intent. A random UUID generated inside the worker isn't: each redelivery would get a fresh identity and bypass the guard. Producers should create the key and include it with a versioned payload.

There is one uncomfortable interval — after a remote side effect succeeds but before the local result is recorded. No generic queue API can erase that ambiguity. Prefer a downstream operation that accepts the same idempotency key. When that is unavailable, use a transactional outbox or a reconciliation process that can compare intended operations with observed downstream state. I'm not sure which option fits your system without knowing whether the side effect and idempotency record share a transactional store; that boundary decides the design.

The following Go contract makes those decisions visible. It is deliberately broker-neutral so the same state machine can sit behind a Node.js adapter in production; the adapter should map its client's delivery, ack, nack, and publish calls to these operations without changing the ordering.

package worker

import (
    "context"
    "errors"
    "fmt"
    "time"
)

var (
    ErrTemporary = errors.New("temporary failure")
    ErrPermanent = errors.New("permanent failure")
)

type Job struct {
    ID             string
    IdempotencyKey string
    Attempt        int
    Payload        []byte
}

type Delivery interface {
    Job() Job
    Ack() error
    NackRetry(delay time.Duration) error
    NackDeadLetter(reason string) error
}

type Claims interface {
    // Claim returns false when this operation already reached a terminal state.
    Claim(ctx context.Context, key string) (bool, error)
    Complete(ctx context.Context, key string) error
    Release(ctx context.Context, key string) error
}

type Handler func(context.Context, Job) error

func ConsumeOne(ctx context.Context, d Delivery, claims Claims, handle Handler) error {
    job := d.Job()
    if job.ID == "" || job.IdempotencyKey == "" {
        return d.NackDeadLetter("invalid envelope")
    }

    claimed, err := claims.Claim(ctx, job.IdempotencyKey)
    if err != nil {
        return err // Leave unacked so broker recovery can redeliver it.
    }
    if !claimed {
        return d.Ack()
    }

    err = handle(ctx, job)
    if err == nil {
        if err := claims.Complete(ctx, job.IdempotencyKey); err != nil {
            return err
        }
        return d.Ack()
    }

    if releaseErr := claims.Release(ctx, job.IdempotencyKey); releaseErr != nil {
        return fmt.Errorf("release claim: %w", releaseErr)
    }
    if errors.Is(err, ErrPermanent) || job.Attempt >= 5 {
        return d.NackDeadLetter(err.Error())
    }

    delay := time.Duration(1<<job.Attempt) * time.Second
    return d.NackRetry(delay)
}

The sample leaves an acknowledgement outstanding if claim storage is unavailable or completion cannot be recorded. That is intentional: there is no durable proof that the job is complete. The catch is that a simple claim implementation needs lease expiry or recovery for a worker that dies while holding a claim. A unique row with only started and complete states can strand work forever. Store an owner token and expiry, or use a transactional inbox whose recovery semantics are documented and tested.

Separate backoff from dead-letter handling

Retries are scheduling decisions, not repeated guesses. Classify failures before you choose a delay. An HTTP 429 Too Many Requests response means the client sent too many requests in a period; the response may include Retry-After, which should take precedence over a locally calculated delay when it is valid. Add jitter to local exponential backoff so many workers don't wake at once.

A malformed payload will not improve after ten minutes. Neither will a domain rule such as "account closed." Mark those failures permanent and dead-letter them immediately with a compact reason, the original job identifier, schema version, attempt count, and timestamps. Avoid copying secrets or an entire sensitive payload into an error string.

Dead-letter storage is a quarantine, not a trash can.

Give it an owner, an age alarm, and a replay procedure. Replay should create a controlled new delivery that preserves the original idempotency key and records who initiated it. If operators edit payloads, record the old and new hashes plus the reason; otherwise a postmortem cannot distinguish recovery from silent data mutation. A bulk replay also needs a rate limit because recovering ten thousand jobs can recreate the dependency pressure that caused the original backlog.

This design is not suitable when a job must coordinate several non-idempotent external systems with one atomic commit. A queue plus a deduplication table cannot provide that transaction. Use a workflow with explicit compensation and reconciliation, or redesign the side effects around idempotent commands. Likewise, stick with an in-process scheduler when work is disposable, finishes within one process lifetime, and losing it on deploy is acceptable; operating a broker, dead-letter path, and replay controls would add failure modes without buying useful durability.

Verify the ugly paths before deployment

The happy-path test proves very little. Run a deterministic fault matrix against a disposable queue and real claim storage. Stop the worker after the side effect but before ack, then confirm that redelivery produces one business effect and one deduplication event. Stop it after claim acquisition, advance the lease clock, and verify recovery. Return HTTP 429 with Retry-After; confirm that the job does not run early. Feed a permanently invalid payload and confirm that it reaches dead-letter storage once rather than bouncing through the active queue.

Also assert invariants, not just log lines:

• One idempotency key has at most one terminal business result.
• Acknowledged deliveries have a durable terminal record.
• Active retries never exceed the configured attempt ceiling.
• Dead-letter entries retain enough metadata for an audited replay.
• Queue age, not queue depth alone, pages the on-call engineer.

Queue depth can rise because producers are busy, and that may be fine. Oldest-message age says work is waiting longer than the service objective. Pair it with completion rate, retry rate by reason, redelivery count, dead-letter age, handler latency, and claim conflicts. Use bounded-cardinality labels: job type and failure class are useful; raw job IDs are not.

Deploy with a small worker cohort and watch age plus duplicate suppression before raising concurrency. More consumers can make a throttled dependency slower, especially when each consumer retries independently. Set concurrency from downstream capacity and observed service time, then keep retry traffic inside the same budget as first attempts.

Roll back without losing ownership

Rollback the consumers first while leaving producers and queue retention intact. Stop accepting new deliveries, let in-flight handlers reach a recorded decision within a shutdown deadline, and close the channel or lease without acknowledging unfinished work. Then deploy the previous worker version and verify that queue age is falling.

Schema compatibility is the usual trap. During a rolling deploy, producers and consumers at adjacent versions coexist. Make consumers tolerate the previous envelope version, and avoid deleting fields until the oldest queued message can no longer contain them. If a release changes idempotency key construction, pause: old and new workers could assign different identities to the same operation. Restore the previous key rule before replaying anything.

Keep rollback boring. The runbook should state the stop order, drain deadline, compatible schema versions, retry ceiling, dead-letter location, and the query that proves one terminal result per key. Do not bulk-requeue while the incident is still producing failures. Stabilize the dependency, sample the quarantine, replay a small batch, and check queue age before continuing.

References

• https://www.rabbitmq.com/docs/confirms
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429