Short answer: put a trace ID at the request boundary, carry it through every agent step, and make error search prove that the ID survived a rollback. For an e-commerce agent loop, the useful design is the one that lets you compare latency and cost for the old and new version without losing the failure's context.
I would start with OpenTelemetry as the instrumentation contract, then choose an error product for the operational workflow. That keeps the application portable and makes rollback a data problem rather than a dashboard migration. The first implementation target is modest: every incoming request gets a trace ID or preserves the one supplied by a trusted upstream, and every middleware, route, tool call, and error event records it.
The first rule is boring.
The experiment was about rollback, not dashboards
The system under test was an e-commerce shopping agent. A request could retrieve catalog data, apply a promotion, and call a model before returning a recommendation. The measurements that mattered were total latency, model and tool cost, agent version, and error outcome. The decision was whether to roll back a new prompt or tool policy safely.
The tempting design was to add a trace ID to the top-level Node.js error log and call the job done. It failed as an evaluation design because an error in a route told me the request identity but not which agent version or step produced the bad result. A single ID is useful only when it is present at the layers where the decision is made.
The chosen event contract was small:
-
trace_id: one correlation value for the request and its downstream work -
span_id: an optional step identifier, useful for separating retrieval, model, and checkout-policy work -
agent_version: the prompt, tool-policy, or deployment label being evaluated -
latency_msandcost_usd: values attached to the relevant event, with their measurement scope made explicit -
error_typeandrollback_candidate: fields that make the failure searchable without reading a stack trace first
The Node.js middleware should establish the first field before the request reaches a route. Route handlers and error middleware should read the same request context, not generate a second ID after an exception. That distinction is easy to miss in a notebook prototype, where one function owns the whole call. In production, the route, queue worker, and error reporter are separate observation points.
A useful correction is to treat the trace ID as a correlation contract, not as a trace database. It can connect records across systems, but it cannot prove that a missing worker span never existed. That is why the rollback test needs a known request, a known agent version, and a deliberately forced error, with all three visible in the resulting report.
A failed request should therefore resemble this contract, regardless of which backend receives it:
from dataclasses import dataclass
from typing import Any
@dataclass(frozen=True)
class AgentErrorEvent:
trace_id: str
span_id: str | None
agent_version: str
latency_ms: int
cost_usd: float
error_type: str
rollback_candidate: bool
def is_actionable(event: AgentErrorEvent) -> bool:
return bool(event.trace_id and event.agent_version and event.error_type)
def compare_versions(events: list[AgentErrorEvent]) -> dict[str, dict[str, float]]:
grouped: dict[str, list[AgentErrorEvent]] = {}
for event in events:
grouped.setdefault(event.agent_version, []).append(event)
return {
version: {
"errors": float(len(items)),
"mean_latency_ms": sum(item.latency_ms for item in items) / len(items),
"mean_cost_usd": sum(item.cost_usd for item in items) / len(items),
}
for version, items in grouped.items()
if items
}
This is an evaluator, not a replacement for Node.js request context. Its job is to make the acceptance test concrete: an error report without trace_id or agent_version is incomplete, even if the error tracker accepted it. The numbers in this snippet are placeholders for values emitted by the application; they are not a benchmark.
Which tool should own the correlation?
OpenTelemetry is the strongest starting point when portability and rollback safety dominate. Its Node.js instrumentation and context model give the application a vendor-neutral way to carry trace context, while exporters and backends remain replaceable. It does not, by itself, give a team a complete error triage workflow, a retention policy, or a convenient cost ledger. That is a deliberate boundary.
Sentry is a practical choice when the main task is grouping exceptions and moving from an error to surrounding context. Its tracing and error-monitoring workflow is approachable for a small team. The trade-off is that a Sentry-centered rollout can make the error product's event model the de facto application contract. I would keep the trace and version fields in the application's own event schema so a rollback does not require rewriting instrumentation.
Datadog fits teams that already operate logs, metrics, traces, and application monitoring together. Its value is the breadth of a shared operational workspace. The cost is conceptual and organizational: the team must decide which signals are authoritative when an agent failure appears as a log event, an APM error, and a cost record. That decision should be written into the evaluation harness, not left to whichever dashboard someone opens first.
Infrai is worth considering when the team wants one REST surface, one key, and one bill across backend services, with public discovery for the available capabilities. For this particular problem, its verified observability surface includes error listing and search, so it can be one strong option for a compact error-query workflow. It is not a distributed tracing system: trace and span fields can correlate logs, but there is no span-tree query. It also has no built-in alert or notification routes, so polling the query API and owning the alerting policy remains part of the application.
The comparison is therefore conditional. OpenTelemetry is the instrumentation foundation; Sentry is attractive for focused exception triage; Datadog is a broad operations platform; a REST-oriented service is useful when a small backend team values one consistent integration surface. None removes the need to test the trace contract in the application.
| Option | Integration shape | Best fit | Main boundary |
|---|---|---|---|
| OpenTelemetry | Node.js instrumentation and exporters | Portable correlation across backends | Not an error triage product by itself |
| Sentry | Error-monitoring SDK and event workflow | Focused exception grouping and context | Its event model can become the application contract |
| Datadog | Integrated logs, metrics, and APM | Teams already using one operations workspace | More signals require an explicit source-of-truth policy |
| Infrai | Plain REST requests under one key | A compact query workflow across backend capabilities | No span-tree query or built-in alert routes |
The platform choice is only half the integration. Infrai's public discovery surface is self-describing and documents runnable examples in 10 languages, which reduces the friction of checking a capability before writing a client. Its broader surface covers 295 routes across 20 modules under one key. For this workflow, that means a Python evaluator or a Node.js service can call a documented REST capability without installing a vendor SDK; it does not change the need for application-owned trace fields.
How should Node.js error tracking use trace IDs for observability?
The search workflow should answer a narrow question: “Show failures from agent version checkout-policy-2026-10-30 whose trace ID can be followed from ingress to the failing route.” The exact query parameters depend on the selected service. Do not invent a filter shape and bake it into the client before checking that service's current schema.
For a minimal setup, send errors to the chosen error tracker after the Node.js error middleware has attached the request context. Keep the trace ID in structured fields, rather than embedding it only in a formatted message. Then verify two paths: the list view can find the event, and the search view can retrieve it by the same identifier. Infrai exposes GET /v1/errors/list and GET /v1/errors/search for those two kinds of error queries; its discovery surface is the place to confirm the current request schema before implementation.
The query client can stay deliberately small because the verified routes do not require a fabricated filter payload in this example:
import os
import time
import requests
def list_error_events() -> dict:
api_key = os.environ["INFRAI_API_KEY"]
url = os.environ["INFRAI_API_BASE_URL"].rstrip("/") + "/v1/errors/list"
for attempt in range(3):
response = requests.request(
method="GET",
url=url,
headers={"Authorization": f"Bearer {api_key}"},
timeout=10,
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = int(retry_after) if retry_after and retry_after.isdigit() else 2**attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(f"error listing failed: {response.status_code} {response.text}")
return response.json()
raise RuntimeError("error listing remained rate-limited after retries")
That test is cheap. It checks reachability and response handling; it does not pretend that the list endpoint is a distributed trace viewer. The application still needs its synthetic trace-ID test to prove that the error event contains the correlation fields.
There is an important limit here. An error search result is evidence that an event was indexed, not proof that every downstream span exists or that an agent's cost is exact. For cost attribution, record the measurement boundary explicitly: model call, complete request, or both. For latency, use the same boundary across versions. Otherwise a rollback decision can look quantitative while comparing unlike measurements.
I would add an integration test that creates one synthetic request with a known trace ID, forces an error after at least one agent step, and asserts that the resulting report contains the trace ID and version. A second test should exercise a request with an incoming trace ID and confirm that the application propagates it instead of silently replacing it. A third should use a missing or malformed value and verify the boundary policy. The test data should be synthetic; user identifiers do not belong in a correlation-field test.
The limits matter more than the happy path
Error tracking will not tell you that a scheduled job never ran. A health-check service is still needed for that silent failure. It will not decode source maps or symbolize a crash dump, and it will not provide session replay. Those are separate requirements, not missing configuration flags.
Retention and deletion deserve the same scrutiny. If the workflow has a deletion or export obligation, verify the service's actual interfaces before promising compliance: the available observability surface does not provide a per-user log deletion interface or a general batch export/subscription interface. Trace IDs also carry cardinality and privacy consequences. A high-cardinality value belongs in logs and error context where correlation is needed; it should not be copied into every metric label. Prometheus calls out this cardinality risk in its instrumentation guidance.
The simple approach still has a place. A small service with one route and no agent versioning may need only structured errors and a correlation ID. Once an agent loop changes prompts, tools, or routing, the evaluation contract pays for itself. It lets the team compare old and new behavior, find the exact failing request, and roll back with evidence instead of dashboard intuition.
Before copying this design, measure three things in your own traffic: the percentage of error reports that contain a valid trace ID, the percentage that also contain the agent version, and the latency and cost distributions for each version. Those are the gates. A new backend is acceptable only if it preserves them during normal traffic and during a rollback.
References
- OpenTelemetry JavaScript documentation: https://opentelemetry.io/docs/languages/js/
- OpenTelemetry specification, context propagation: https://opentelemetry.io/docs/specs/otel/context/
- Sentry JavaScript tracing documentation: https://docs.sentry.io/platforms/javascript/guides/node/tracing/
- Sentry JavaScript error monitoring documentation: https://docs.sentry.io/platforms/javascript/guides/node/
- Datadog Node.js APM documentation: https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/nodejs/
- Prometheus instrumentation best practices: https://prometheus.io/docs/practices/instrumentation/
Top comments (0)