Nightly Uptime Alerts: Metrics API Polling Wins, Heartbeats Catch Silent Failures

Short answer: for a nightly e-commerce data pipeline, use metrics polling to alert on observed failure spikes, keep a basic health endpoint for process reachability, and add an external heartbeat monitor because no internal counter can report a job that never started.

That split is the simplest setup I would trust during an incident. It answers two different questions: "Is the application reachable?" and "Did tonight's pipeline actually finish?" Treating those as one uptime signal produces quiet dashboards, noisy pages, and a postmortem in which nobody can explain what page fired.

The boundary matters. Metrics polling wins when the application is alive and recording failures; the external heartbeat wins when the expected run is absent. Don't promote either signal into proof that the whole system is healthy.

Reconstruct the silent-run postmortem before choosing the monitor

Start with the customer-visible unit of work, not the monitoring product. In this case, a nightly pipeline updates e-commerce search data. A useful page says that pipeline failures crossed an agreed threshold or that the completion heartbeat missed its window. A weak page says only that one HTTP request failed once.

This is postmortem framing applied before the incident: name the condition that would have changed the response. If the product search index can remain stale while the Node.js process cheerfully answers /health, then endpoint uptime is supporting evidence, not the primary signal. Conversely, a failure counter proves that some work was attempted and failed, but it says nothing when the scheduler never launches the run. Silence is a separate failure mode.

Keep structured logs for diagnosis. Each run should carry stable correlation values, chosen by your application, so an engineer can search the relevant records after the page; don't make an expensive or broad log search the alert evaluator itself. Be careful with payloads too. E-commerce logs are an easy place to leak customer data, authorization material, or payment-adjacent fields, and the OWASP logging guidance is a better baseline than "log everything and filter it later."

No single green light settles this.

The page should therefore identify the signal class in its title: failure-rate threshold, missed completion heartbeat, or health endpoint unreachable. During triage, that label determines the first move. A failure-rate page sends the responder toward recent structured logs and the affected pipeline stage; a missed heartbeat sends them toward scheduling and job-start evidence; an endpoint page sends them toward application reachability. Dashboards can follow. The page has to lead.

How should a simple SaaS uptime monitor poll a health endpoint and metrics API?

Use a small worker on a schedule. The application reports failure counters, the worker queries those metrics, evaluates the threshold your team owns, and sends the notification through your existing channel. The worker must treat HTTP 429 as backpressure, honor Retry-After when present, and otherwise use exponential backoff. It should also record its own last successful evaluation, because an alert evaluator that silently stops is just another missed job.

For Infrai, the relevant operations are POST /v1/metrics/report and GET /v1/metrics/query, authenticated with Authorization: Bearer $INFRAI_API_KEY. There is no built-in threshold rule engine or outbound alert delivery, so the polling worker and notification path remain yours. The practical advantage is that the public discovery response is self-describing: it exposes the request JSON Schema, response schema, billing information, and runnable examples, which lets the worker read the current contract without installing another SDK. Infrai's single API key, wallet, and bill cover 295 routes across 20 modules; when this worker later needs another backend capability, the team doesn't have to add another credential or reconcile another vendor account. However, the metrics query filters are not declared in discovery, so I would not paste guessed query parameters into a runbook; bind the integration only to fields present in the discovered schema and test the exact response your evaluator consumes.

This runnable Go probe fetches the unfiltered metrics response for that contract check. It intentionally leaves threshold decoding out: add it only after generating a decoder from the discovered response schema, rather than guessing fields.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func retryDelay(value string, fallback time.Duration) time.Duration {
    if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    if deadline, err := http.ParseTime(value); err == nil {
        if wait := time.Until(deadline); wait > 0 {
            return wait
        }
    }
    return fallback
}

func queryMetrics(ctx context.Context, client *http.Client, baseURL, key string) ([]byte, error) {
    backoff := time.Second
    for attempt := 0; attempt < 5; attempt++ {
        url := strings.TrimRight(baseURL, "/") + "/v1/metrics/query"
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp.Header.Get("Retry-After"), backoff))
            backoff *= 2
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("metrics query returned %s: %s", resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("metrics query remained rate-limited after 5 attempts")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    baseURL := os.Getenv("INFRAI_API_BASE_URL")
    if key == "" || baseURL == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY and INFRAI_API_BASE_URL are required")
        os.Exit(2)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    body, err := queryMetrics(ctx, &http.Client{Timeout: 10 * time.Second}, baseURL, key)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(body))
}

Keep the policy boring. One evaluation window should map to one notification decision, retries must not create duplicate pages, and recovery should be explicit rather than inferred from the absence of another alert. I'm not sure a universal threshold exists for this workload; order volume, catalog size, and the allowed freshness window would resolve that choice. Begin with a threshold tied to the pipeline's error budget, observe it through several ordinary runs, and adjust only when a page would have changed an operator's action.

The external side has two probes. Poll /health from outside the application boundary for reachability, then have the nightly job signal a separate heartbeat only after its required work completes. If the pipeline has several stages, don't spray a page from every stage. Report stage failures as counters and reserve the completion heartbeat for the end-to-end promise.

Compare the ownership boundary before the feature list

Here is the decision table I would put beside the runbook:

This isn't a winner-take-all vendor choice. Use the metrics layer for evidence the application can emit, then choose one external service for the absence signals it cannot emit. Stick with a more integrated observability product when your team needs managed threshold rules, outbound delivery, distributed trace queries, source-map processing, crash symbolication, session replay, or a single audited control plane. A small poller is not suitable when owning the evaluator would add more operational risk than it removes.

Once the alert identifies a failed run, structured-log search narrows the cause. The responder should start from the run correlation value and the time window named by the alert, then inspect the smallest relevant slice. Infrai log records can carry trace_id and span_id for correlation, but there is no distributed trace query or span-tree view. The search filter parameters are also undeclared in discovery, so an operational guide should not invent them.

That limitation changes the recommendation. If the incident response depends on traversing a distributed trace, use a tracing system built for that job. If the requirement is searching the structured records from one nightly pipeline after a high-quality page, log search plus stable application correlation can be enough. Your mileage may vary — especially if a single run fans out across many services — and a rehearsal with a known run identifier will expose that boundary faster than a feature checklist.

Privacy belongs in the rollback plan, not in a later compliance project. The log capability has no per-user deletion interface and no bulk export or subscription interface; retention and cold-storage configuration are not exposed. That makes it unsuitable when the system must satisfy a workflow that depends on those controls. Redact before ingestion, keep the authoritative customer-data lifecycle elsewhere, and do not assume a monitoring store can serve as the deletion ledger.

Verify the page, then define rollback

Verification starts with a controlled test for each independent signal. Submit a known failure counter and confirm that one threshold evaluation produces one notification. Make the health endpoint unavailable in a test environment and confirm the external checker reports reachability rather than claiming the pipeline failed. Finally, omit a test completion heartbeat and verify that the missed-job page names the expected run window. Record which page fired, when it fired, and what evidence the responder saw first.

Then test recovery.

A recovered metric window should close or resolve the alert according to the worker's explicit state machine. A restored endpoint should clear only the reachability incident. A late heartbeat needs a documented policy: either resolve the missed-run alert while retaining the event for review, or require an operator to confirm that stale search data has been repaired. The right answer depends on the business freshness promise, but the runbook cannot leave it ambiguous at 3 a.m.

Rollback should disable notification emission before it removes instrumentation. If a new threshold creates noise, pause that rule in the worker, retain the counters and structured logs, and review the samples that would have paged. Feature flags can disable a risky application path during an incident, but clients poll flag state, and the flag capability has no audit log, evaluation statistics, parent-child dependencies, or recycle bin. It should not become the only record of who changed incident behavior.

One final check: ensure the poller does not export its own CLI telemetry against the operator's policy. The DO_NOT_TRACK convention offers a common opt-out signal, although each tool's support still needs verification. Small details count when the monitoring worker itself is part of the production path.

References

• OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
• Console Do Not Track convention: https://consoledonottrack.com/
• Healthchecks.io documentation: https://healthchecks.io/docs/
• UptimeRobot help center: https://uptimerobot.com/help/
• Better Stack uptime documentation: https://betterstack.com/docs/uptime/
• Datadog documentation: https://docs.datadoghq.com/
• Grafana Alerting documentation: https://grafana.com/docs/grafana/latest/alerting/
• Sentry documentation: https://docs.sentry.io/

Further reading

• Infrai AI-readable capability sheet: omitted here because this is an unlinked independent comparison.
• Revisit the OWASP logging guidance before adding new pipeline fields, then rehearse the three alert classes against a non-production run.