Short answer: at-least-once delivery makes a reminder safer to retry, but unsafe to execute blindly. A standard queue may deliver the same renewal-reminder job more than once, so its impact on reliability is determined by the notification effect: make it idempotent, record enough state to recover after a deadline, and retain only the evidence needed to explain a decision to a user.
For an e-commerce system, the concrete job is easy to state: send a reminder after a merchant-defined business deadline, such as 09:00 on the next working day. The difficult part is deciding what happens after a timeout, a worker restart, or a successful send followed by a lost acknowledgement. The queue's promise is delivery of work, not exactly-once execution of the side effect.
How does at-least-once delivery in standard queues affect reliability?
It changes the default failure decision. If the worker cannot prove that processing finished, the recovery-friendly action is to make the job visible again and try it. That can produce two sends. Suppressing the retry can produce zero sends. For a renewal reminder, the second outcome is usually harder to discover, while the first is immediately visible to a customer. The design therefore needs a duplicate-safe boundary around the send operation.
I start with a stable reminder identity, not a generated attempt ID. A useful key might combine the account, renewal obligation, and reminder window: acct_4821:renewal_2026-11-03:business-deadline. Every delivery attempt carries that key. The notification service stores an outcome for the key, and a retry checks that outcome before creating another external effect. A concurrency-safe insert or compare-and-set is important here; a read followed by a write leaves a race between two workers.
This does not make the queue exactly once. It makes the business operation tolerant of at-least-once delivery. That distinction belongs in the runbook and in the test names.
The retry is intentional.
Which state should survive a retry?
The durable record should separate scheduling from sending. It needs the intended deadline, the current status, the stable idempotency key, and timestamps for the first attempt, latest attempt, and final outcome. It should also capture a small reason code such as sent, already_sent, not_due, or permanent_failure. Do not use the queue message as the only audit trail: acknowledgement and retention policies can erase the very evidence needed during recovery.
A worker can then follow a narrow state transition:
- Load the reminder by its stable identity.
- Confirm that the business deadline has passed in the configured timezone and that the renewal is still eligible.
- Atomically claim the send or observe that another worker already claimed it.
- Attempt the delivery.
- Persist the outcome before acknowledging the queue message when the surrounding system allows that ordering.
The final ordering deserves a test of its own. If the worker acknowledges first and crashes before recording success, the next delivery may send again. If it records success first and crashes before acknowledgement, the next delivery should find sent and become a no-op. The latter still requires the send boundary itself to support an idempotency key; otherwise the crash can happen after the external send but before the local record.
How should a business deadline behave during recovery?
A deadline is a policy, not merely a timestamp. A reminder due at 09:00 should not be treated as eligible at 08:59 because a worker is eager, and a retry at 09:07 should not be discarded merely because the original queue attempt failed. Store the timezone and the calendar rule used to calculate the deadline. The Linux crontab(5) specification is a useful reminder that scheduled work is expressed in calendar terms, while the delivery system still has to handle missed or repeated executions.
For each retry, recalculate eligibility from durable business data. Do not trust a stale payload to decide that a renewal still exists, that a customer still needs the reminder, or that the deadline has not changed. A cancellation between enqueue and execution should become a recorded no-op, not a sent message. A temporary delivery failure should remain retryable with backoff. A malformed address or an explicitly rejected destination needs a terminal outcome and an alert path.
There is a practical limit: idempotency can prevent duplicate effects only when the effect has a durable identity or a provider boundary that accepts one. If the final channel cannot deduplicate, the system must choose between duplicate risk and missed-message risk, then make that choice visible in product policy. No queue setting removes that trade-off.
What does the observability bill actually measure?
The dominant cost is often not the queue record itself. It is the volume of logs, retained payloads, and high-cardinality labels generated by every attempt. A single reminder with 5 attempts can create 5 verbose payload logs, while a short outcome record can explain the same event with one identity, one status, and a handful of timestamps. Retention multiplies that difference across every account and every retry.
I keep attempt count and latency, but I do not keep full customer payloads in routine logs. The stable reminder key is useful for correlation; putting account IDs, email addresses, campaign names, and arbitrary error strings into metric labels turns each value into another time-series dimension. Keep those details in access-controlled event storage when investigation requires them, and sample successful no-op attempts more aggressively than permanent failures.
The trade-off is real. Less retained detail makes a rare incident slower to reconstruct. More retained detail increases storage, search cost, and exposure of personal data. A workable policy might retain aggregate counters for the reporting period, compact outcome events for the operational window, and a sampled trace for successful paths. The exact duration is a governance decision; the engineering requirement is that it covers the longest plausible recovery and dispute window.
A useful dashboard has counts for due, sent, already-sent, delayed, retryable, and terminal outcomes. It also has a duplicate-prevention counter. That counter should not be mistaken for duplicate sends: it measures attempts stopped by the idempotency record. The distinction tells an operator whether the queue is redelivering normally or whether the external channel is producing an effect twice. For a user reminder, that difference determines whether an operator can explain a duplicate attempt without claiming that a duplicate message was sent.
How do you test recovery without testing only the happy path?
The test matrix should force the worker to stop at each boundary: before the claim, after the claim, after the external send, after the local outcome write, and before acknowledgement. Re-deliver the same message after each stop. The expected result is one customer-visible reminder and a clear durable outcome, except where the chosen channel cannot provide a deduplication boundary.
Use a fake clock for the business deadline and a controllable delivery adapter for timeouts. Test a daylight-saving transition, a changed renewal status, two workers racing for one key, and a queue redelivery after a process restart. Test the operational path too: can an operator find the stable identity, distinguish a delayed reminder from a suppressed duplicate, and safely replay only a terminally recoverable event?
The deployment rule is modest: change the state machine and the worker together, then observe duplicate-prevention and terminal-failure rates before widening traffic. Alerts should be tied to missed deadlines and growing retry age, not to raw attempt volume alone. Retries are expected; unbounded retries are not.
At-least-once delivery is a useful reliability contract because it favors recovery from uncertainty. It becomes a customer problem when a system treats every delivery as permission to perform the side effect again. For renewal reminders, the durable identity, deadline recheck, idempotent send boundary, and deliberately small evidence trail form one design. Keep the retry, but make the repeated execution harmless.
That is the whole contract.
Further reading
- Linux
crontab(5)manual page: https://man7.org/linux/man-pages/man5/crontab.5.html
Top comments (0)