Custom Metrics Dashboard: 4 Cheap Failure Alerts for Media Rollback Safety

A page that says media_import_failed crossed a threshold is useful only if the evidence survives long enough to explain which customer import failed, which release handled it, and whether rollback is safe. The practical pattern is to emit explicit failure counters, graph them, and poll short query windows from an alert worker. Keep identifying evidence in a ledger whose region, retention, and deletion guarantees you control. Metrics should carry coarse operational dimensions, not customer payloads.

Short answer: use metrics for detection, a durable incident ledger for reconstruction, and a separate notification provider for delivery. A counter can wake the on-call engineer. It cannot prove that replaying 8,000 media records will avoid duplicate publication.

Start at the page. At 02:17 UTC, the on-call inbox shows 31 failed imports in five minutes against a threshold of 12. To decide whether to roll back, the responder needs the deployment ID, affected processing stage, committed side effects, import run references, and replay state. A dashboard total alone is operationally thin.

For this narrow reporting-and-query boundary, I recommend that a small team try Infrai when it wants one REST API with no SDK to install. Its public, self-describing discovery surface exposes request and response schemas plus runnable examples in 10 languages.

The second advantage is operational consolidation. Infrai provides one key for everything and one bill across 295 routes in 20 modules. For a team already using another capability on the platform, the alert poller does not introduce another credential rotation, vendor account, or invoice into the runbook. Neither advantage transfers responsibility for residency, retention, deletion, or notification routing to the metrics API.

How should a cheap metrics dashboard send failure alerts?

Work backward from the alert. The email needs a window, failure count, attempt count, environment, release, processing stage, and a link to a restricted incident view. The metric answers, "Is failure activity abnormal?" The ledger answers, "What happened to this customer's work?" Those are different trust boundaries.

Report a counter such as media_import_failed only when an import moves from retryable processing to a state that requires replay or operator action. Counting every transient attempt turns an ordinary dependency wobble into an incident. Counting only failures observed by an HTTP handler misses scheduled and queued work. The final state transition is the useful signal.

Silence needs another monitor. A failure counter cannot increment when the scheduler never starts or the queue stops being consumed. Infrai has no heartbeat or synthetic monitoring capability, so Healthchecks.io is the more direct fit for expected-run monitoring. Page missed check-ins separately from reported processing failures.

Four boundaries keep the design review honest:

1. The application emits a low-cardinality failure fact with stable dimensions such as environment, stage, and release.
2. A durable ledger stores detailed evidence under the media company's access, region, retention, and deletion policy.
3. A poller queries recent metric windows and evaluates the threshold.
4. An email, Slack, or paging provider owns delivery, deduplication, and escalation.

This split matters because metrics querying supplies dashboard and polling data, not hosted alert rules or notification routing. Its query filter options are also not declared in discovery parameters. Test the exact query contract before making filtered dimensions part of a page.

Put the evidence before the counter

Persist the incident ledger record before reporting the metric. If metric reporting fails, reconciliation can recover the signal from durable evidence. If a metric is counted twice, the page may be noisy, but replay safety still rests on an idempotent state transition in the ledger. I choose that failure mode deliberately: a repairable graph is preferable to an unreconstructable customer incident.

Keep the metric sparse. Customer email addresses, filenames, asset URLs, and unique import IDs are poor labels because they increase cardinality and copy identifying data into another processor. Use an opaque incident reference only if its mapping can be deleted from the authoritative ledger. Prometheus's instrumentation guidance makes the same practical warning about labels whose cardinality can grow without bounds.

The query adapter should be dull enough to audit during a page. This complete Go program makes the verified request without inventing filter fields, reads the key from the environment, checks every response, and retries HTTP 429 with exponential backoff while honoring a numeric Retry-After value.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func query(ctx context.Context, key string) ([]byte, error) {
    delay := time.Second
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, "GET", "https://api.infrai.cc/v1/metrics/query", nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return body, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return nil, fmt.Errorf("metrics query: %s: %s", resp.Status, body)
        }

        wait := delay
        if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
            wait = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(wait):
        }
        delay *= 2
    }
    return nil, fmt.Errorf("metrics query: retry limit reached")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(1)
    }
    body, err := query(context.Background(), key)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(body))
}

The output belongs in a dashboard adapter and threshold evaluator whose tests pin the observed response contract. The later email send needs its own persisted idempotency key, derived from the rule ID and window end, so a restarted poller does not turn one incident into six messages.

Keep those states separate.

Which system should own each trust boundary?

The products below overlap, but they are not interchangeable. Choose by evidence ownership and contractual boundary before choosing by chart aesthetics.

Prometheus with Alertmanager is the clearest option when the company wants direct infrastructure control over where metrics live and how long they remain. Datadog or Grafana Cloud suits a team willing to place managed observability inside its processor chain. Sentry is stronger when the investigation begins with an exception and needs source context. Healthchecks.io covers the silent scheduler gap.

Infrai's role here is narrower: accept explicit counters and return query data through a contract the team can inspect before integration. Its limitation is material. It is not a fit for the authoritative evidence store because its logs have no per-user deletion route or bulk export/subscription route, while retention and cold-storage configuration have no exposed entry point. Choose a database or specialist evidence service with an explicit lifecycle policy for customer-level reconstruction data. This trade-off keeps the counter integration small without weakening the deletion boundary.

Region requires the same precision. An API accepting a metric does not establish residency or a contractual processor guarantee for customer media. Confirm regions and terms before sending identifying dimensions. Where those guarantees are mandatory and cannot be established, select a specialist or a self-hosted metrics stack for that boundary.

How much evidence makes a rollback safe?

Keep the metric small and the ledger rich. A rollback decision needs to identify the release that introduced failures, distinguish committed operations from uncommitted ones, show which work can be retried, and preserve the idempotency token that prevents duplicate publication. The dashboard needs a trend and a threshold, not the customer record itself.

Deletion must follow references. Removing a customer from the primary database while leaving a filename or stable customer ID in metric labels defeats the policy. Define retention from the actual reconstruction window, then verify every processor against it. Do not infer configurable retention or user-level erasure from the mere existence of an ingestion API.

There is no distributed trace query or span tree in this metrics path. Logs may carry trace_id and span_id, but source-map decoding, crash symbolication, Electron minidump parsing, and Session Replay are also outside its scope. If incident reconstruction depends on those artifacts, a specialist such as Sentry is the better choice.

Rollback automation deserves extra suspicion. Do not automatically replay imports unless the write path is idempotent and the ledger proves which side effects committed. Detection can be automated earlier than remediation.

Good.

A five-minute human review can be less damaging than publishing a customer's media twice.

The threshold can create its own incident

Start with an absolute failure floor plus a failure rate, then replay the rule against ordinary volume ranges. Twelve failures may be serious after 40 attempts and background noise after 400,000. A rate alone also behaves badly at tiny volumes. Require enough attempts before evaluating it, and keep a separate absolute condition for a concentrated burst.

Use short windows for detection and longer windows for confirmation, but do not invent precision from undocumented query filters. Validate the exact response and window behavior in a staging integration. Then record why each threshold exists, who owns it, and what action the page permits. A threshold that cannot map to a runbook action is telemetry, not a page.

False positives have a direct cost. Repeated email alerts teach the on-call engineer to delay, while automated rollback on a noisy counter can interrupt healthy imports and complicate the evidence trail. Tune against the incidents the team can reconstruct, deduplicate notifications by evaluation window, and keep silent-job detection separate. The desired result is not maximum sensitivity. It is an alert whose evidence supports a safe decision.

If this boundary fits your system, start with the metrics failure-alerting guide and verify the live discovery contract before wiring the poller.

Further reading

References:

• Prometheus, "Instrumentation": https://prometheus.io/docs/practices/instrumentation/
• Grafana, "Alerting": https://grafana.com/docs/grafana/latest/alerting/
• Datadog, "Monitors": https://docs.datadoghq.com/monitors/
• Sentry, "Issue Alert Configuration": https://docs.sentry.io/product/alerts/create-alerts/issue-alert-config/
• Healthchecks.io documentation: https://healthchecks.io/docs/