DEV Community

delay.run
delay.run

Posted on Edited on

How to replay failed webhooks safely: store the raw request, then re-run it with a 40-line Python script

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

Sooner or later this happens: a deploy breaks your webhook handler for a few hours. The provider retries for a while, then gives up. You fix the bug, and now there's a gap. Payments that happened, subscriptions that changed, orders that shipped, and your database knows nothing about them.

Some providers let you resend events from their dashboard or CLI, one by one or in bulk. That's the first thing to check. But not all do, retention windows are often short, and resending through the provider means waiting for their delivery queue.

The more reliable approach is to keep your own copy of every webhook request, and replay from it. This post shows the table and a small replay script.

Step 1: store every webhook before processing it

The trick is to separate receiving from processing. The HTTP handler does as little as possible: verify the signature, store the raw request, return 200.

CREATE TABLE webhook_inbox (
  id           bigserial PRIMARY KEY,
  provider     text        NOT NULL,
  event_id     text        NOT NULL,
  event_type   text        NOT NULL,
  received_at  timestamptz NOT NULL DEFAULT now(),
  headers      jsonb       NOT NULL,
  body         bytea       NOT NULL,      -- raw bytes, exactly as received
  status       text        NOT NULL DEFAULT 'pending',  -- pending | done | failed
  attempts     int         NOT NULL DEFAULT 0,
  last_error   text,
  UNIQUE (provider, event_id)
);
Enter fullscreen mode Exit fullscreen mode

Store the raw bytes of the body, not parsed JSON. Signature verification works on raw bytes, and you want to be able to re-verify later if you need to prove an event is genuine.

The UNIQUE (provider, event_id) constraint gives you free deduplication of provider retries: insert with ON CONFLICT DO NOTHING.

A separate worker (or a short delayed job per event) reads pending rows, runs the business logic, and marks them done or failed with the error.

Step 2: make the processing function callable directly

The business logic should be a plain function that takes the stored row, not something buried in an HTTP route:

def process_event(row) -> None:
    event = json.loads(bytes(row["body"]))
    handler = HANDLERS[event["type"]]   # e.g. "invoice.paid" -> handle_invoice_paid
    handler(event)
Enter fullscreen mode Exit fullscreen mode

This is what makes replays easy and safe: the replay script calls exactly the same function the worker does. No re-signing, no fake HTTP requests, no special code path.

Step 3: the replay script

#!/usr/bin/env python3
"""Replay failed or unprocessed webhooks from webhook_inbox."""
import argparse, time, traceback
import psycopg
from psycopg.rows import dict_row
from app.webhooks import process_event   # the same function the worker uses

p = argparse.ArgumentParser()
p.add_argument("--since", required=True, help="e.g. 2026-11-16T10:00Z")
p.add_argument("--until", required=True)
p.add_argument("--status", default="failed", choices=["failed", "pending"])
p.add_argument("--type", help="only this event type")
p.add_argument("--rate", type=float, default=5.0, help="events per second")
p.add_argument("--apply", action="store_true", help="actually process (default: dry run)")
args = p.parse_args()

sql = """SELECT * FROM webhook_inbox
         WHERE status = %(status)s AND received_at >= %(since)s AND received_at < %(until)s
           AND (%(type)s::text IS NULL OR event_type = %(type)s)
         ORDER BY received_at, id"""

with psycopg.connect(autocommit=True, row_factory=dict_row) as conn:
    rows = conn.execute(sql, vars(args)).fetchall()
    print(f"{len(rows)} event(s) to replay{'' if args.apply else ' (dry run)'}")
    ok = failed = 0
    for row in rows:
        label = f"{row['id']} {row['event_type']} {row['event_id']}"
        if not args.apply:
            print("would replay", label); continue
        try:
            with conn.transaction():
                process_event(row)
                conn.execute("UPDATE webhook_inbox SET status='done', attempts=attempts+1, "
                             "last_error=NULL WHERE id=%s", [row["id"]])
            ok += 1; print("ok    ", label)
        except Exception:
            failed += 1
            conn.execute("UPDATE webhook_inbox SET attempts=attempts+1, last_error=%s "
                         "WHERE id=%s", [traceback.format_exc()[-2000:], row["id"]])
            print("FAILED", label)
        time.sleep(1 / args.rate)
    print(f"done: {ok} ok, {failed} failed")
Enter fullscreen mode Exit fullscreen mode

Run it in two steps:

python3 replay.py --since 2026-11-16T10:00Z --until 2026-11-16T14:30Z
python3 replay.py --since 2026-11-16T10:00Z --until 2026-11-16T14:30Z --apply --rate 2
Enter fullscreen mode Exit fullscreen mode

Why each piece is there

Dry run by default. The first run just lists what would be replayed. You'll often spot something surprising, like 4,000 events when you expected 400, before anything happens.

Ordered by received_at. Replaying in roughly the original order reduces surprises, but don't rely on it; webhooks can arrive out of order in the first place. Handlers should already be safe in any order.

One transaction per event. The connection runs in autocommit mode, so each conn.transaction() block is its own transaction. The business logic and the status update commit together. If processing fails, nothing is half-applied, and the row stays failed with the new error.

Rate limit. Handlers often call the provider's API or send emails. Replaying 2,000 events as fast as possible can hit rate limits and create a second incident.

Same function as production. A replay path that differs from the normal path is a replay path that has never been tested.

Replays are only safe if the handler is idempotent and you can see every attempt. delay.run works well for the processing half: one delayed HTTP job per stored event, retried with backoff, with a run log, so a failed event retries on its own instead of waiting for a manual replay.

Before you replay, check idempotency

Replaying assumes that processing an event twice is harmless. Make sure:

  • Every handler checks whether it has already applied the event (a processed-events table, or a unique constraint on the business record).
  • Side effects like emails are keyed so they don't go out twice.
  • "Add" operations (credits, seats, balances) are written as "set" operations, or guarded by the event ID.

If any handler isn't idempotent, replay that event type separately, with a careful look at each row.

Retention

Keep the inbox for at least as long as your longest plausible incident plus investigation time; 30 to 90 days is common. The bodies contain customer data, so apply the same retention and access rules as the rest of your database, and drop old rows on a schedule.

Related reading

Do you keep raw copies of incoming webhooks? I'm curious how many teams replay from their own inbox versus relying on the provider's resend button.

Top comments (1)

Collapse
 
acemeridian profile image
Ace Meridian •

I ran your table and replay.py as written against PostgreSQL 16.4 with psycopg 3.3.9, using a toy credit.added handler that inserts into a ledger table. With 5 failed rows the dry run listed 5, --apply replayed 5, and each event got exactly one ledger row. Two cases broke it, though.

  1. The handler's writes are only rolled back if they go through the same conn. process_event(row) takes no connection, so a handler that uses the app's own connection or pool commits on its own. I made evt_3 insert its ledger row and then raise (say, an email provider 503). The script marked it failed, but the credit stayed. After the bug was fixed, running the same replay again credited it a second time:
FAILED 3 credit.added evt_3
done: 4 ok, 1 failed
ledger rows per event: [('evt_1', 1), ('evt_2', 1), ('evt_3', 1), ('evt_4', 1), ('evt_5', 1)]
ok     3 credit.added evt_3
ledger rows per event: [('evt_1', 1), ('evt_2', 1), ('evt_3', 2), ('evt_4', 1), ('evt_5', 1)]
Enter fullscreen mode Exit fullscreen mode
  1. Two people running --apply at the same time both replay every row, because the SELECT happens once up front and nothing claims a row. With a 0.2 s handler, both runs printed done: 5 ok, 0 failed, and every event ended with attempts 2 and 2 ledger rows. The same thing can happen with --status pending if the normal worker is still running.

I fixed both by passing conn into process_event(row, conn) and re-reading each row inside the transaction:

with conn.transaction():
    row = conn.execute(
        "SELECT * FROM webhook_inbox WHERE id = %s AND status = %s "
        "FOR UPDATE SKIP LOCKED", [id_, args.status]).fetchone()
    if row is None:
        skipped += 1; continue
    process_event(row, conn)
    conn.execute("UPDATE webhook_inbox SET status='done', attempts=attempts+1, "
                 "last_error=NULL WHERE id=%s", [id_])
Enter fullscreen mode Exit fullscreen mode

After that change, evt_3's failed attempt left 0 ledger rows and the retry left 1. The two concurrent runs split the work (2 ok + 3 skipped, and 3 ok + 2 skipped), so each event got 1 row.

One smaller thing: --since 2026-11-16T10:00 without a Z is read in the session's TimeZone. On my machine (Asia/Dubai) that window matched 0 rows, when the UTC window matched 5. It's worth requiring the offset in argparse.

Do your handlers get the connection passed in, or do they open their own?