GodfreySterling9226

Posted on Oct 26

Healthtech Feature Metrics Dashboard Backend — Choose API Charts and Evidence Logs

#observability #node #healthtech

Use metrics for operational charts, but keep a separate, deliberately small event trail for incident reconstruction. TL;DR: a counter can tell a healthtech team that appointment confirmations dropped; it cannot reliably establish which workflow version ran, what outcome occurred, or which internal correlation key joins the steps. Trying to make one telemetry stream do both jobs either destroys useful context or turns every chart query into a log scan.

The deciding constraint is signal quality versus noise. A small team needs fast KPI time series, yet an incident review needs enough evidence to follow one customer interaction without copying health data into telemetry. Those needs overlap. They are not identical.

Should a feature metrics dashboard backend use an API or logs?

A dashboard question is aggregate: how many scheduling attempts succeeded per five-minute bucket, by workflow version? An incident question is selective: what happened to one attempt, in what order, under which release and feature state?

Metrics fit the first shape. Their dimensions need bounded value sets because each distinct attribute combination creates another time series. Customer IDs, appointment IDs, and free-form error messages do not belong there. They create noise, and they turn identifiers into query dimensions when the chart only needed workflow_version and outcome.

Noise wins fast.

Logs or structured events fit the second shape because each record can preserve ordered facts. They also carry more privacy risk. GDPR Article 5 requires purpose limitation, data minimization, and storage no longer than necessary. So "log everything, just in case" is a poor incident strategy even before query speed enters the discussion.

The split is strict:

Need	Record	Useful fields	Keep out
KPI chart	Counter or histogram	route class, outcome, workflow version	customer and appointment identifiers
Reconstruction	Structured event	timestamp, pseudonymous correlation key, step, outcome, release, feature state	names, symptoms, free text, raw request bodies
Debug detail	Error event	error class, safe code, correlation key	stack data containing request values

This is less convenient than emitting one giant object. Good. Convenience at ingestion can become permanent exposure at rest.

The constraint that changed the design

Feature state matters during a reconstruction. A rollout can put two customers through different code paths at the same wall-clock time. Martin Fowler's feature-toggle guidance distinguishes toggle categories and recommends exposing current toggle configuration so operators can understand behavior. The practical observability consequence is narrow: record the evaluated feature state or a versioned configuration fingerprint with the event. Do not infer it later from today's configuration.

That does not justify dumping an entire flag document into every record. For a confirmation workflow, a compact variant value such as control or reminder_v2, plus the deployed release, answers the relevant question. The permitted values should be reviewed like an API schema.

I benchmark the query shapes, not vendor demos. The test corpus should include the expected event rate, the longest allowed retention window, and a burst that resembles a retry storm. Measure p50 and p95 for a fixed dashboard query and for a single-correlation reconstruction. Also count bytes written and distinct metric series. A result without those four measurements hides the likely failure mode.

Do not use invented production numbers as a target. Set thresholds from the admin dashboard's latency budget and the incident response objective, then test the exact queries the team will run.

The smallest working TypeScript boundary

The first version needs 2 interfaces and 1 policy function. No sprawling configuration file. This example deliberately accepts enumerated fields and constructs the emitted objects; callers cannot smuggle arbitrary request data through a spread operator.

type Outcome = "started" | "confirmed" | "rejected";
type Variant = "control" | "reminder_v2";

interface MetricPoint {
  name: "appointment_confirmation_total";
  value: 1;
  attributes: {
    outcome: Outcome;
    workflowVersion: "v3";
  };
}

interface EvidenceEvent {
  occurredAt: string;
  correlationKey: string;
  step: "request_received" | "eligibility_checked" | "message_queued";
  outcome: Outcome;
  release: string;
  variant: Variant;
}

interface TelemetrySink {
  count(point: MetricPoint): Promise<void>;
  append(event: EvidenceEvent): Promise<void>;
}

async function recordWorkflowStep(
  sink: TelemetrySink,
  input: EvidenceEvent,
): Promise<void> {
  await Promise.all([
    sink.count({
      name: "appointment_confirmation_total",
      value: 1,
      attributes: {
        outcome: input.outcome,
        workflowVersion: "v3",
      },
    }),
    sink.append({
      occurredAt: input.occurredAt,
      correlationKey: input.correlationKey,
      step: input.step,
      outcome: input.outcome,
      release: input.release,
      variant: input.variant,
    }),
  ]);
}

The sink is intentionally generic. An adapter can translate the counter to a standards-based metrics exporter and append the event to an access-controlled store. The domain code does not know a backend name, an endpoint, or a query language. Time-to-first-call stays short: implement 2 methods, inject 1 object, and move on.

Keep it boring.

There is one sharp edge. Promise.all makes telemetry failure visible to the caller, which may be correct for evidence that is part of a regulated operational process but wrong on a patient-facing request path. Decide explicitly. A bounded queue with backpressure, retry limits, and a dead-letter path can decouple transport failure; it also creates another store that needs access controls, deletion rules, and monitoring. Hiding the decision inside an SDK is worse.

Test the contract with schema-invalid fields, sink timeouts, duplicate delivery, and out-of-order events. The reconstruction query must tolerate retries, so every production event should gain a stable event identifier before durable ingestion. That identifier is evidence plumbing, not a chart dimension.

Deployment checks that catch noisy telemetry

Ship the event schema and dashboard query with the feature. A canary should compare the expected workflow transitions with emitted counters and events. Feature toggles complicate that check: exercise every live variant, then verify that the variant appears in reconstruction records while chart dimensions remain bounded.

Three alerts are enough for the initial pipeline: rejected records, queue saturation, and a sustained mismatch between terminal workflow events and the corresponding counter. Alerting on every application error reproduces the noise problem upstream.

Privacy review belongs in the release path too. Document the purpose of every event field, who can query it, its retention rule, and the deletion behavior. GDPR Article 32 calls for measures appropriate to risk, including capabilities around confidentiality, integrity, availability, and resilience. Encryption and access control matter, but neither makes unnecessary fields necessary.

What I would change at scale

At higher volume, I would separate transport from retention. The application emits the same typed contract into a bounded local or regional collector; downstream processors create chart aggregates and write the narrow evidence stream under different access and retention policies. Batch export reduces call overhead, while explicit queue limits stop observability traffic from consuming unlimited memory.

I would also version schemas and validate compatibility in CI. Additive optional fields are easier to roll out than renamed meanings. A consumer that reconstructs an incident must be able to explain an older record after the application has moved on.

There is a real trade-off. Dual paths add adapters, health checks, and reconciliation work. A logs-only prototype has fewer moving parts, and it may be adequate when volume is low and chart latency is not important. A metrics-only prototype is faster to chart, but it cannot retain per-interaction evidence without abusing high-cardinality labels. Move to the split when both aggregate operations and individual reconstruction are actual requirements, not hypothetical ones.

The final decision rule is compact: chart bounded aggregates; retain only the minimum structured events that can reconstruct the customer workflow; join them through a pseudonymous key available only to authorized incident tooling. Benchmark both query shapes against your own latency objectives. Keep sensitive payloads out of both.

DEV Community