Designing Internal Service Status from Metrics and Logs (For AI Agent Loops)

An uptime page for a media AI agent loop has a stricter constraint than a general monitoring console: an editor needs to know whether a publishing job can finish correctly, while an operator needs enough evidence to reconstruct why it did not. Displaying every metric and log line satisfies neither need. The right first version is a small, read-only projection built from bounded metrics, correlated terminal events, and an explicit freshness rule.

TL;DR: model the loop as a state machine, expose one aggregate snapshot to the internal admin UI, and keep raw logs behind drill-down links. Show availability, end-to-end latency, error ratio, and an estimated usage unit for each workflow class. Mark stale data as unknown rather than healthy. This preserves the distinction between service reachability and completed editorial work, which is where many simple status pages become misleading.

How should a Node.js uptime dashboard combine metrics and logs?

A media workflow may accept a story brief, retrieve background material, call one or more models, run a policy check, and persist a draft. A process-level health check can succeed while that sequence repeatedly times out or produces no publishable result. For an internal page, the unit of health should therefore be the completed workflow, not the HTTP process.

Start with four questions derived from the latency, traffic, errors, and saturation signals described in the Google SRE monitoring chapter: are workflows finishing, how long do they take, how much work is arriving, and is a constrained dependency accumulating pressure? The dashboard can present those questions without pretending that one green badge captures them all.

For each workflow class, such as article-draft or caption-rewrite, define a terminal success event and a terminal failure event. Require both to carry the same opaque run identifier, policy version, and workflow class. The identifier is the audit key. It also makes retries visible: a retry is another attempt attached to the same logical run, not a second successful publication.

Exactly-once processing is not something a dashboard can confer on the underlying pipeline. Its projection can still adopt an exactly-once mindset. Deduplicate terminal events by (run_id, attempt_id, terminal_state), reject contradictory duplicates from the aggregate, and retain enough evidence for reconciliation. If a success counter says 100 while the terminal-event ledger contains 99 unique successful runs, the page should report an integrity discrepancy instead of averaging the disagreement away.

Retries lie.

Unknown matters.

A snapshot older than the agreed collection interval is not evidence of availability. It is unknown, with its observation timestamp shown plainly. That single rule prevents a frozen collector from leaving a reassuring green panel on screen.

Derive the projection before choosing storage

The status view needs a compact contract. The example below uses a 5-minute window for fast operational feedback and a 24-hour window for context; those are example policy choices, not universal thresholds. The response separates observed values from decisions so a later threshold change does not rewrite history. A Node.js admin application can consume the same JSON contract even when the aggregation worker is written in another language.

package status

import (
    "encoding/json"
    "net/http"
    "time"
)

type Window struct {
    DurationSeconds int64   `json:"duration_seconds"`
    Completed       uint64  `json:"completed"`
    Failed          uint64  `json:"failed"`
    P95LatencyMs    float64 `json:"p95_latency_ms"`
    UsageUnits      float64 `json:"usage_units"`
}

type Snapshot struct {
    Workflow      string    `json:"workflow"`
    Status        string    `json:"status"`
    ObservedAt    time.Time `json:"observed_at"`
    FreshUntil    time.Time `json:"fresh_until"`
    Recent        Window    `json:"recent"`
    Daily         Window    `json:"daily"`
    IntegrityGaps uint64    `json:"integrity_gaps"`
}

type SnapshotReader interface {
    Read(r *http.Request) (Snapshot, error)
}

func Handler(reader SnapshotReader, now func() time.Time) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        snapshot, err := reader.Read(r)
        if err != nil {
            http.Error(w, "status snapshot unavailable", http.StatusServiceUnavailable)
            return
        }
        if now().After(snapshot.FreshUntil) {
            snapshot.Status = "unknown"
        }
        w.Header().Set("Content-Type", "application/json")
        w.Header().Set("Cache-Control", "no-store")
        if err := json.NewEncoder(w).Encode(snapshot); err != nil {
            return
        }
    })
}

UsageUnits should use the stable unit the pipeline actually records, not a currency estimate silently mixed across model contracts. Currency conversion belongs in a separately versioned rule with its effective time and source. This is the same discipline a ledger projection needs: preserve the immutable observation, then apply a reviewable interpretation.

The aggregate store may be a relational table, a time-series database, or an in-memory cache backed by durable events. The status contract does not depend on that choice. What does matter is atomic publication: calculate a complete snapshot off to the side, validate its counts and timestamps, then replace the visible version in one operation. Readers should never see the five-minute latency from one calculation paired with the daily failure count from another.

This design has a clear limitation: a precomputed projection favors predictable reads and auditable states over open-ended exploration. It is not suitable as a replacement for a metrics explorer or restricted log-search system. The trade-off is deliberate because an incident-facing status page must remain cheap to read and easy to reconcile, while deeper diagnosis can tolerate a separate, slower path with tighter access controls.

Join metrics and logs without turning the page into a log viewer

Metrics answer trend questions efficiently. Logs explain individual runs. Joining them on the page is useful only if the join is deliberate: an aggregate row links to a filtered set of terminal events for the same workflow and time window, and an individual run links to its attempts. Do not ship raw prompts, generated copy, credentials, or personal data into a broad admin view. Retention and access must follow the organization's compliance policy, and access to the status page should itself leave an audit record.

The useful event is small. A terminal record needs the run and attempt identifiers, workflow class, terminal state, start and finish timestamps, policy version, usage unit, and a bounded error category. Free-form error text belongs in restricted logs, not in a label or aggregate key. Unbounded labels create noise and make the metric series harder to reason about.

This division also controls ingestion volume. Log services commonly charge according to ingestion volume; the CloudWatch pricing page, for example, documents per-GB log ingestion dimensions. The architectural response is not to make price the main criterion. It is to emit fewer, higher-value events, keep high-cardinality evidence out of metrics, and choose retention according to investigation and compliance needs.

A compact admin table can then carry the information operators actually compare:

These values are illustrative data, not benchmark claims. Notice that zero traffic is not automatically an outage, and stale observation is not converted into zero. The integrity column also remains separate from runtime failure: a workflow may finish while its evidence trail is incomplete.

Set decisions that resist alert noise

A dashboard and an alert have different jobs. The page can show every supported workflow class; a page-worthy alert should represent an actionable breach sustained for a defined window. One failed run may deserve a visible count and a drill-down link without waking anyone. Conversely, an integrity gap may require immediate investigation even while the completion ratio looks normal.

Use a decision table owned in version control. For example, a team might classify a workflow as degraded when its error ratio breaches a reviewed threshold in two consecutive five-minute windows, or when its p95 latency exceeds a workflow-specific objective with enough completed samples to make the percentile meaningful. The exact threshold must come from the service objective and traffic profile. No source can supply it generically.

Three distinctions keep the signal honest:

1. Separate unknown from healthy; missing telemetry cannot prove success.
2. Separate logical runs from attempts; retries must not inflate throughput or erase the original failure.
3. Separate operational errors from audit-integrity gaps; each has a different owner and response.

Test those rules with deterministic fixtures before testing the HTML. Feed the projector duplicate terminal events, late arrivals, a retry that succeeds, contradictory terminal states, an empty window, and a snapshot beyond fresh_until. Assert both the aggregate and the retained reconciliation record. Then load-test the read path independently, because an internal page that runs expensive ad hoc log queries can compete with the collection path during the incident it is meant to explain.

Roll out the view without losing audit continuity

Begin in shadow mode. Produce snapshots, compare their completed and failed counts with the terminal-event ledger, and publish the discrepancies only to the implementation team. Once reconciliation remains within the team's explicit acceptance rule, expose the page read-only to operators; alerting can follow after the display has earned trust.

During migration, version the snapshot schema and keep the prior projector available until consumers have moved. Deploy decision-rule changes with an effective timestamp, record who approved them, and annotate the page when a rule changes. A latency graph that crosses a newly changed threshold is then explainable rather than mysterious.

The final acceptance test is intentionally plain: given a run identifier, an operator can trace every attempt to one terminal outcome, reproduce the aggregate row for its window, and explain the displayed state from a versioned rule. If any link is missing, the page is decoration, not an operational control.

Sources

• Google SRE Book, "Monitoring Distributed Systems": https://sre.google/sre-book/monitoring-distributed-systems/
• Amazon CloudWatch pricing: https://aws.amazon.com/cloudwatch/pricing/