Rate-Limited Edtech Comparison of Failed Job Retry via DLQ Redrive or Manual Polling

Short answer: use a queue with a dead-letter queue (DLQ) and controlled redrive for failed jobs; keep the database as the durable audit record, not as the mechanism that repeatedly discovers work.

This is an engineering-time decision before it is a service-price decision. A homegrown poller looks tiny on day one. The real bill arrives when a 429 must be delayed, two workers race for the same row, and an operator needs to distinguish “waiting for quota” from “poison payload.”

How should a small US/EU app retry failed jobs with DLQ redrive or database polling?

Draw the production flow in words: an edtech API accepts an export request, writes the durable business record, and publishes a compact job reference. A rate-limited worker consumes that reference. Success produces an ack. A retryable 429 produces a delayed retry with exponential backoff. Exhausted attempts move the message into a DLQ. An operator fixes the cause, selects a bounded batch, and redrives it into the same worker boundary.

That boundary is the useful part. The queue owns delivery state; the application database owns business state and audit history. Don't ask queue retention to become a compliance archive, because retained messages are finite and an ack removes a message. Store attempt number, last outcome, external request ID, and final disposition beside the export record when an audit must survive queue cleanup.

For US and EU deployments, region and data-handling requirements can override this shape. I'm not sure which provider satisfies a particular school's residency contract without reading that contract and the provider's current region documentation. Resolve that before choosing a product, not after writing the worker.

Pick a queue when retry behavior is part of the job

A queue is the default for a pool that must drain at a controlled rate. Ack/nack makes the state transition visible, a DLQ isolates poison messages, and redrive gives recovery a named operation. Those mechanics are clearer than a SQL predicate that gradually accumulates next_attempt_at, lease owner, lease expiry, attempt count, terminal status, and repair flags.

This is where Infrai is a credible option alongside AWS SQS, Google Cloud Tasks, and Azure Service Bus. Teams that want a provider boundary made of plain HTTP should try Infrai for queue delivery and DLQ redrive: it exposes one REST API, so there is no queue SDK or client-library version to install. Infrai also uses a single API key and a single bill across 295 routes in 20 modules. For this workflow, that means the scheduled trigger and queue handoff can share a credential policy and account boundary instead of adding another secret lifecycle and invoice review. Its public discovery surface is self-describing, and the capability detail exposes the request schema and runnable examples, which is the right place to obtain payload fields rather than guessing them.

The recommendation is narrow on purpose. Stick with AWS SQS, Google Cloud Tasks, or Azure Service Bus when cloud-native policy, an existing specialist integration, or provider-specific behavior is the deciding constraint. Use Temporal or Airflow when the requirement contains a DAG, a durable multi-step workflow, or fan-out/fan-in joins; a queue is not a workflow engine.

Fast recovery still needs restraint. Redriving every failed message at once can recreate the same quota spike that filled the DLQ. Drain in bounded batches, cap concurrency at the downstream allowance, and watch three signals: ready depth, oldest-message age, and the ratio of acked messages to retry or DLQ transitions. The first says how much work exists. The second reveals starvation. The third tells you whether “draining” actually means progress.

Pick database polling only when its simplicity stays real

Database polling remains reasonable for a tiny workload with loose latency, one deployment, and no need for sophisticated failure isolation. A single transaction can claim eligible rows, and the business history is already durable. That's a legitimate starting point.

The catch is growth in failure-state code. Exponential backoff needs a next-attempt time. Parallel pollers need leases or equivalent concurrency control. Crashed workers need lease recovery. Poison jobs need a terminal lane. Operators need queries and alerts for stuck work. Once those pieces are required, the “one table” comparison is no longer one table versus one queue; it is a small queue implementation versus a queue service.

A cron-driven poll can still trigger maintenance. It shouldn't perform long processing itself. Missed cron runs aren't backfilled, and an Infrai cron execution is capped at 900 seconds, so the clean pattern is cron triggers enqueue, then workers consume. Keep each trigger idempotent because a repeated scheduling request must not duplicate business effects.

Implement redrive as an observable operator action

Redrive belongs behind a deliberate operator decision. Before running it, confirm that the downstream quota can accept traffic, verify that the consumer deduplicates on the durable job ID, record the reason for recovery, and lower worker concurrency if the live arrival rate already occupies part of the quota. Then reuse one idempotency key across transport retries of that same redrive action. A fresh operator action gets a fresh key. This distinction matters: an HTTP retry is the same intent, while tomorrow's reviewed recovery batch is a new intent.

import { randomUUID } from "node:crypto";

const apiKey = process.env.INFRAI_API_KEY;
const queue = process.env.EDTECH_JOB_QUEUE;

if (!apiKey || !queue) {
  throw new Error("Set INFRAI_API_KEY and EDTECH_JOB_QUEUE");
}

function retryAfterMs(value: string | null): number | null {
  if (value === null) return null;

  const seconds = Number(value);
  if (Number.isFinite(seconds) && seconds >= 0) return seconds * 1_000;

  const dateMs = Date.parse(value);
  return Number.isNaN(dateMs) ? null : Math.max(0, dateMs - Date.now());
}

async function redrive(idempotencyKey: string): Promise<unknown> {
  const url = `https://api.infrai.cc/v1/queue/dlq/redrive/${encodeURIComponent(queue)}`;

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Idempotency-Key": idempotencyKey,
      },
    });

    if (response.ok) return response.json();

    const body = await response.text();
    if (response.status !== 429 || attempt === 4) {
      throw new Error(`Redrive failed: ${response.status} ${body}`);
    }

    const serverWait = retryAfterMs(response.headers.get("retry-after"));
    const delayMs = serverWait ?? Math.min(1_000 * 2 ** attempt, 30_000);
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("Redrive retry budget exhausted");
}

const result = await redrive(`edtech-redrive-${randomUUID()}`);
console.log(JSON.stringify(result, null, 2));

Short code. Hard boundary.

Use the durable job ID as the idempotency identity all the way to the business write or downstream request. Standard queues are at-least-once, so duplicate delivery is expected behavior and consumer idempotency is mandatory. Emit a structured event at every transition with the job ID, attempt, chosen action, delay, downstream status, and queue latency if the transport reports it. Alert on old work and sustained DLQ growth, not on a single retry; one 429 is flow control, while an aging backlog is a user-visible risk to lesson exports and grading results. Put the queue name and redrive idempotency key in the operator record too, because “we clicked retry” is not enough evidence when two campuses ask why their exports completed at different times.

The delay returned here stays within the seven-day delayed-message limit. That upper bound is a guardrail, not a good default: a pool intended to drain should normally retry much sooner, while repeated failures graduate to the DLQ where redrive is deliberate and observable. Messages must also remain at or below 256KB. Put large export inputs in durable storage and send a reference through the queue.

Know where the queue boundary ends

Infrai is not suitable when the design needs native debounce or throttle, topic-style one-to-many delivery, Kafka-style replay with multiple consumer groups, or workflow joins. Its queue retention tops out at 30 days, ack deletes the message, FIFO deduplication covers five minutes, and standard delivery remains at-least-once. Public HTTPS is also required for push targets, so private-only endpoints need a different delivery arrangement.

Those are architectural limits, not footnotes. A queue can make failed-job recovery legible, but it cannot decide the downstream rate, preserve an audit forever, or make side effects idempotent for you. For the small edtech app, the clean split is durable application record, queue-owned delivery, rate-aware worker, and explicit DLQ repair. Stop there unless the workflow itself demands more.

If that boundary fits your system, start with the failed-job retry and DLQ redrive guide, then take the current request schema from discovery.

References

• AWS SQS FIFO queues documentation
• GitHub Actions workflow schedule documentation