Message Queues vs Direct HTTP for Reminder Delivery — A Go Runbook Explained

Use a queue when reminder delivery can outlive the request that created it, then drain that queue with an explicit rate limit, a bounded worker pool, and a retry policy that preserves the delivery SLO. Direct HTTP is fine for a small, synchronous action. It becomes a poor control surface when a B2B SaaS system has a large reminder cohort, a rate-limited downstream, or a maintenance event that requires thousands of jobs to move at once.

The important distinction is not “HTTP versus queues” as a technology preference. It is ownership of time. An HTTP caller owns the connection, timeout, and immediate result. A queue owns the work after admission, so the caller can return while workers regulate concurrency and make failure visible. That separation usually costs some delivery latency and operational machinery. In exchange, it prevents a reminder burst from turning every web request into a downstream retry storm.

Why are message queues preferred to direct HTTP calls here?

Suppose a SaaS account schedules 200,000 user reminders for a campaign. A direct design loops over recipients and calls a notification service from the API process. The API now carries business traffic, connection management, downstream throttling, and retry decisions in one execution path. A five-second dependency timeout is not a capacity plan; it is merely a place for one request to stop waiting.

With a queue, the scheduling transaction records an intent and publishes a job containing a stable reminder ID, recipient reference, scheduled time, and idempotency key. A worker later claims that job and performs the delivery. The user-facing request can report that the work was accepted, while an operations view reports whether delivery is keeping its SLO. Those are different promises and should have different measurements. The trade-off is explicit: the caller gets less immediate certainty, while the platform gains a durable place to apply backpressure, retry work, and inspect age. I would accept that extra moving part only when the latency and failure isolation it buys matter more than synchronous simplicity.

The queue is the boundary.

This does not make delivery reliable by magic. A queue can duplicate messages, delay them, or fill faster than workers can drain them. It gives the system a place to measure those conditions and a lever to respond. The failure moves from “the API request is stuck” to “backlog age is rising,” which is a much more useful signal.

The common mistake is to acknowledge a job before the durable handoff has completed, then call the design asynchronous. The handoff must be durable enough for the stated promise. If the promise is only “we accepted your request,” a database record may be enough. If the promise is “the reminder will be attempted,” the record and queue publication need an atomicity strategy, such as an outbox consumed by a publisher. Choose the promise first.

The drain runbook for a rate-limited worker pool

Treat the downstream limit as a hard capacity boundary. If the provider allows 100 requests per second, starting 100 workers does not guarantee safe operation: retries, uneven job duration, multiple tenants, and clock skew all consume the same budget. I start with a lower effective rate, observe the response distribution, and raise it only when the error budget and queue-age target remain healthy.

The useful capacity calculation is deliberately boring:

drain rate = worker concurrency / average service time

If 20 workers each take 400 ms on average, the idealized rate is 50 jobs per second. Real capacity is lower when the downstream returns throttles, when retries occupy workers, or when jobs serialize on a per-tenant limit. Plan from the slow percentile and reserve headroom for ordinary traffic. A one-minute burst should not consume the entire hour's error budget.

The worker should enforce four boundaries. These are deliberate controls, not knobs to tune until the graphs look smooth:

1. A global concurrency limit for the downstream dependency.
2. A per-tenant or per-destination limit when fairness is part of the product contract.
3. A deadline for one attempt, separate from the overall retry window.
4. A maximum number of attempts, after which the job becomes an explicit failure for review or replay.

Here is a minimal Go worker loop. It uses a generic queue interface so the control logic does not depend on a particular broker. The example assumes Ack removes a successfully completed message and Nack makes an unsuccessful one eligible for a later attempt; a production adapter must document those semantics precisely.

package main

import (
    "context"
    "errors"
    "log"
    "time"
)

type Reminder struct {
    ID             string
    IdempotencyKey string
    TenantID       string
}

type Queue interface {
    Receive(context.Context) (Reminder, error)
    Ack(context.Context, Reminder) error
    Nack(context.Context, Reminder, time.Duration) error
}

type Sender interface {
    Send(context.Context, Reminder) error
}

func run(ctx context.Context, q Queue, sender Sender, concurrency int) error {
    if concurrency < 1 {
        return errors.New("concurrency must be positive")
    }

    sem := make(chan struct{}, concurrency)
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
        }

        reminder, err := q.Receive(ctx)
        if err != nil {
            return err
        }

        sem <- struct{}{}
        go func(r Reminder) {
            defer func() { <-sem }()

            attemptCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
            defer cancel()

            if err := sender.Send(attemptCtx, r); err != nil {
                if nackErr := q.Nack(ctx, r, 30*time.Second); nackErr != nil {
                    log.Printf("nack reminder %s: %v", r.ID, nackErr)
                }
                return
            }

            if err := q.Ack(ctx, r); err != nil {
                log.Printf("ack reminder %s: %v", r.ID, err)
            }
        }(reminder)
    }
}

The loop is intentionally incomplete around rate tokens, shutdown draining, and poison-message handling. Those are policy decisions, not details to hide behind a worker count. The sender must also accept the idempotency key, or the downstream operation must be naturally idempotent. Otherwise a timeout after the provider accepted the reminder creates the worst possible ambiguity: retrying may duplicate it, while abandoning it may lose it.

A graceful deployment stops receiving new work, lets active attempts finish until their deadline, and leaves unprocessed messages available for another worker. A process that exits after receiving but before acknowledging must not silently discard that message. Test that transition with a real broker or a faithful adapter; a unit test that calls a function directly will miss it. I would also test the less obvious sequence in which the downstream accepts the request, the worker loses its connection before Ack, and the queue redelivers the job. That is where an idempotency key stops being a nice field in a schema and becomes the difference between one reminder and two.

What should the SLO measure?

Measure from the reminder's scheduled time, not just from enqueue time. For a user-facing reminder, useful signals include delivery-attempt latency, queue age, oldest-message age, attempt success rate, duplicate rate, downstream throttle rate, and dead-letter count. The queue depth is useful but incomplete: ten thousand fast jobs and ten old jobs have very different customer impact.

Define the service level in terms the product can explain. For example: “99% of reminders receive a delivery attempt within five minutes of their scheduled time, excluding cancellations.” The exact target belongs to the product contract, not to the broker. Alert on burn rate or sustained age, and include tenant-level views so one noisy account cannot hide behind a healthy global average.

Capacity planning should use arrival rate, service time, retry load, and the permitted backlog age. If jobs arrive faster than the sustainable drain rate, increasing concurrency only moves pressure to the dependency. The corrective actions are to reduce admission, defer optional work, negotiate a larger limit, or accept a longer SLO. There is no safe autoscaling policy that can manufacture downstream capacity.

When is direct HTTP still the right choice?

Direct HTTP is often the better design when the caller needs the result before responding, the work is short, the dependency's capacity is comfortably above peak demand, and a failed attempt can be shown to the caller without losing business state. Adding a queue to every request creates a second system to operate, plus eventual-consistency behavior that the product may not need.

The queue earns its place when the work is bursty, retryable, independently observable, or bounded by a dependency that the request path should not control. For reminders, those conditions are common: schedules create synchronized bursts, users expect the system to tolerate a temporary provider failure, and a rate-limited worker pool needs a shared control point.

Here is the buy-versus-build boundary I use when choosing the mechanism. The names are examples of different operational shapes, not endorsements.

The selection should follow the delivery contract. If the only requirement is reliable handoff from a transaction, an outbox may be enough. If independent consumers need replayable history, a log-like system may fit better. If the team cannot staff broker operations, a managed service can reduce on-call work while increasing dependency and lock-in concerns. I would record that trade-off in the architecture decision record instead of pretending the choice is free.

How do you verify a drain before trusting it?

Start with a replayable test dataset: include cancelled reminders, duplicate keys, tenants with different limits, a downstream timeout, a throttling response, and a worker restart after receipt but before acknowledgement. Verify that a retry does not create an unintended duplicate and that a poison job reaches a reviewable terminal state. Test scheduled-time ordering separately from queue-order behavior; those are not automatically the same guarantee.

For a controlled rollout, cap the worker pool, enable a kill switch that pauses admission, and expose the oldest scheduled reminder age. Send a small cohort through the new path, compare attempt latency and duplicate rate with the existing path, then expand only if the downstream throttle rate and error-budget burn remain within the agreed bounds. Rollback should pause new publication or route new jobs to the previous consumer while preserving already durable jobs for a deliberate replay decision.

The operational question is simple: if the queue is full or the downstream is unavailable for 30 minutes, what does the customer see, and who decides what happens next? Write that answer before launch. A queue is valuable because it makes that decision explicit; it is dangerous when it merely hides an unbounded pile of work.

References

• https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows
• https://www.rfc-editor.org/rfc/rfc9110
• https://www.rfc-editor.org/rfc/rfc6585
• https://www.rabbitmq.com/docs/confirms
• https://kafka.apache.org/documentation/#intro_concepts_and_terms
• https://docs.nats.io/nats-concepts/jetstream
• https://microservices.io/patterns/data/transactional-outbox.html