Event Notifications Batch Email SMS Partial Failure Queue Troubleshooting

Short answer: treat a bulk event notification as independently durable recipient tasks, then expose terminal status for every email and SMS task; a queue is healthy only when troubleshooting can show which recipient is waiting, which failed, and which message is safe to retry.

For an order-receipt service in a developer-tools product, that audit trail matters more than squeezing another message into a batch. The integration decision starts with failure containment.

Start with the constraint: one receipt, many delivery attempts

The useful unit is not the API call that accepted a batch. It is the recipient task created after payment settles. Give it an immutable event ID, a recipient ID, a channel, a template revision, an attempt count, and timestamps for queued, sent, and terminal states. A batch record can summarize progress, but it must not replace those child records.

This model makes partial failure explicit. If 997 email tasks reach a provider and three are rejected, the batch is neither a total success nor a total failure. The three tasks need a reason, a retry policy, and a way to prove that a second attempt will not create a duplicate receipt. Use an idempotency key derived from the order event and recipient, not from the transient batch request.

I once started debugging a queue that appeared stuck because its dashboard counted only completed batches. I've seen this mislead an on-call engineer for hours: the workers were processing messages; one poison task kept being re-delivered, and the aggregate counter never advanced. The fix was boring: lease each recipient task, record the lease expiry, and make the batch view a projection of child status. Boring is good here.

The counter is not the queue.

What should event notifications reveal about batch email SMS partial failures?

Polling is an observation tool, not a repair mechanism. A status view should return counts by state and a cursor to recipient-level records. Operators need to distinguish queued, in-flight, delivered, permanently rejected, and retryable states, plus the last provider response category and the next attempt time. Keep the response bounded; a million-recipient batch should not require downloading a million rows just to see that a queue is advancing.

A practical polling loop uses a monotonic snapshot time and a short backoff. If the number of in-flight tasks is unchanged across several snapshots, inspect lease expiry and worker heartbeats before adding more workers. If only one channel is flat, isolate that channel's concurrency and rate limits instead of declaring the whole event pipeline unhealthy. Your mileage may vary when provider delivery receipts arrive late, so call a task delivered only after the provider's documented terminal signal, not merely after an HTTP acceptance response.

The queue should also expose age percentiles for pending tasks. A single oldest-task gauge catches a poison message; a p95 age catches broad slowdown. Those two numbers answer different questions, and combining them hides both failure modes.

For troubleshooting, preserve the state transition evidence instead of overwriting it with the latest status. A recipient can move from queued to in-flight, time out, return to retry-wait, and finally become delivered or permanently failed; each transition should carry an event timestamp, worker identity, attempt number, and a bounded reason code. That history lets an operator separate a genuinely stuck queue from a healthy queue whose downstream acknowledgements are slow, and it gives support a precise answer when a customer asks why one receipt arrived twice while another never arrived. It also makes replay reviewable: before re-enqueuing a task, compare its idempotency key, template revision, and payload hash with the original event, then record who approved the replay. Without that chain, “bulk send completed” is an attractive but empty statement.

Retry only states classified as transient. A timeout, connection reset, or explicit throttling response can be retried with bounded exponential backoff and jitter. An invalid address, an unsubscribed recipient, or a malformed payload should become a terminal rejection that a human can fix or suppress. Never retry a permanent rejection merely because the batch total looks disappointing.

Template rendering belongs before the task is leased to a worker. Store the template revision and rendered payload hash with the task, while keeping secrets out of logs. Mustache's documented sections and escaping rules are a useful baseline for predictable rendering, but the notification system still needs tests for missing variables and channel-specific length limits.

from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class RecipientTask:
    event_id: str
    recipient_id: str
    channel: str
    template_revision: str
    state: str = "queued"
    attempts: int = 0
    next_attempt_at: datetime | None = None

def retryable(task: RecipientTask, category: str, now: datetime) -> bool:
    if category not in {"timeout", "connection_reset", "throttled"}:
        task.state = "permanent_failure"
        return False
    task.attempts += 1
    if task.attempts > 5:
        task.state = "permanent_failure"
        return False
    task.next_attempt_at = now + timedelta(seconds=2 ** task.attempts)
    task.state = "retry_wait"
    return True

That five-attempt ceiling is an example policy, not a universal constant. The important property is that the limit is visible, configurable, and included in the runbook.

Compare designs by failure containment, not send speed

A provider-managed batch can reduce integration work, while an app-owned queue gives you finer control over leases, replay, and per-recipient evidence. A hybrid design can submit small provider batches but still persist one local task per recipient. Choose based on which failure you can tolerate and investigate.

The catch is operational ownership: an app-owned queue is not suitable when your team cannot monitor leases, dead letters, and template changes. Stick with a simpler provider workflow when a receipt can be regenerated safely and per-recipient investigation is not a product requirement.

Email also has a policy boundary. Yahoo's sender guidance emphasizes authentication, reputation, and unsubscribe handling; a queue that reports “accepted” without monitoring those signals can still produce a poor delivery outcome. SMS has its own consent, regional, and throughput constraints, so do not copy email retry assumptions into the SMS worker.

Roll out a queue you can explain

Start with shadow records: create recipient tasks and status transitions while the existing sender remains authoritative. Compare counts, rendering hashes, and terminal reasons for a representative slice. Then enable one channel, one template revision, and a bounded percentage of traffic. Keep a replay command that requires an explicit event ID and recipient ID; “retry the batch” is too blunt for a payment receipt.

Before widening the rollout, rehearse three cases: a single permanent rejection, a provider timeout storm, and a worker that stops renewing leases. The success criterion is not a green dashboard. It is a short answer to “what happened to this recipient, and what will happen next?”

References

• https://mustache.github.io/mustache.5.html
• https://senders.yahooinc.com/best-practices/