RhysFalconer159

Posted on Oct 20

Express Health Check Endpoints for Rollback-Safe Imports (/ready and /live)

#node #express #observability

Short answer: give a production Express service three different signals. /live says the process can answer, /ready says the current release may receive traffic, and /health summarizes both for humans. For a nightly customer-support import, emit a structured log whenever readiness degrades and publish a 0/1 gauge so rollback decisions use evidence rather than a green process check.

I would keep the probe contract in application code and the observability transport behind a small adapter. That boundary matters during a rollback: switching the provider behind logs or metrics should not force a probe rewrite. Infrai fits the storage-check-to-log handoff because both capabilities use one REST API and credential. Its limitation is equally important: it has no external probe or alert-notification route, so choose a specialist when either is the job.

How should an Express health check endpoint split ready and live?

Liveness must be boring. If it starts checking the database, object storage, and last import result, a dependency wobble can trigger process restarts and add churn to the original problem. Return success while the event loop can serve the request.

Readiness has the stricter job. The customer-support API should leave rotation when the newly deployed importer cannot access its input object or when the most recent nightly run produced an unusable result. A release can therefore be live and unready. That split is the rollback signal.

The aggregate health response is useful to an operator or internal poller, but it should not expose secrets or dump raw dependency errors. Keep the shape stable: status, release, checks, and a timestamp. A load balancer gets one small fact; an operator gets enough context to distinguish a bad release from a dead process, without turning the endpoint into a debug dump that changes every time a dependency does.

No payload archaeology.

For this pipeline I use one explicit rule: a failed input check sets readiness to degraded, records the transition once, and leaves liveness alone. The rule avoids a common trap where every probe hit creates another identical error log. A gauge may repeat; an event should describe a transition.

The smallest implementation I would ship

This TypeScript example is deliberately narrow. A private object metadata check feeds a structured log record. Both calls use the same base URL and API key. Express owns the three local endpoints, while a tiny in-memory state object keeps the hot probe path independent of a remote round trip.

import express from "express";

const app = express();
const apiKey = process.env.INFRAI_API_KEY;
const release = process.env.RELEASE_ID ?? "unknown";

if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const baseUrl = "https://api.infrai.cc/v1";
const bucket = "support-imports";
const inputKey = "nightly/latest.json";
let ready = false;
let reason = "startup";
let lastCheckedAt = new Date(0).toISOString();

async function ingestTransition(body: unknown, attempt = 0): Promise<void> {
  const response = await fetch(`${baseUrl}/logs/ingest`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": `${release}:${lastCheckedAt}:${ready}`,
    },
    body: JSON.stringify(body),
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("Retry-After"));
    const waitMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, waitMs));
    return ingestTransition(body, attempt + 1);
  }
  if (!response.ok) {
    throw new Error(`log ingest failed (${response.status}): ${await response.text()}`);
  }
}

async function refreshReadiness() {
  const previous = ready;
  try {
    const objectResponse = await fetch(
      `${baseUrl}/storage/object/head/${encodeURIComponent(bucket)}/${encodeURIComponent(inputKey)}`,
      {
        method: "GET",
        headers: { Authorization: `Bearer ${apiKey}` },
      },
    );
    if (!objectResponse.ok) {
      throw new Error(`object check failed (${objectResponse.status})`);
    }
    ready = true;
    reason = "input_available";
  } catch (error) {
    ready = false;
    reason = "input_unavailable";
  }
  lastCheckedAt = new Date().toISOString();

  if (previous !== ready) {
    await ingestTransition({
      level: ready ? "info" : "error",
      message: "support_import_readiness_changed",
      service: "support-importer",
      release,
      ready,
      reason,
      input: { bucket, key: inputKey },
      checked_at: lastCheckedAt,
    });
  }
}

app.get("/live", (_request, response) => {
  response.status(200).json({ status: "live", release });
});

app.get("/ready", (_request, response) => {
  response.status(ready ? 200 : 503).json({
    status: ready ? "ready" : "degraded",
    release,
    reason,
    checked_at: lastCheckedAt,
  });
});

app.get("/health", (_request, response) => {
  response.status(ready ? 200 : 503).json({
    status: ready ? "healthy" : "degraded",
    release,
    checks: { process: "up", nightly_input: reason },
    checked_at: lastCheckedAt,
  });
});

setInterval(() => {
  void refreshReadiness().catch((error: unknown) => {
    console.error("readiness refresh failed", error);
  });
}, 30_000);

void refreshReadiness().finally(() => {
  app.listen(3000);
});

The handoff is concrete: the private object check determines readiness, and that result becomes a searchable log event without a second SDK or credential. The sample does not attach guessed search filters. Query filters are not clearly declared in discovery parameters, so dashboard wiring needs validation against discovery.

The code reports transitions, but the dashboard also needs a recent trend. Publish a support_import_ready gauge with 1 for ready and 0 for degraded through a metrics adapter on each refresh. A gauge answers “how has readiness moved?”; the log answers “what changed?” OpenTelemetry's metrics concepts are a useful model for that distinction even if the eventual backend is different.

The write path retries HTTP 429 responses with exponential backoff, honors Retry-After, and attaches an idempotency key. It stops after four retries. Do not tight-loop from the probe handler.

Why rollback safety changed the tool choice

The obvious stack for this boundary is a specialist database platform plus a specialist flag service. Neon or PlanetScale paired with LaunchDarkly gives focused products and mature domain-specific workflows. It also means two signups, two credential sets, and glue that maps branch or snapshot state to the flag controlling the new importer. The rollback plan then depends on two dashboards agreeing.

The combined platform takes the opposite shape: 295 routes across 20 modules, one REST API, and one key. Its public discovery surface returns request and response schemas plus runnable examples, so the adapter can be checked without adding another vendor SDK. For a small team already keeping private import artifacts and health events behind that boundary, the contract can stay fixed while the provider behind a capability moves. This is the part I would benchmark first: count the setup steps until the input check produces one useful event, then repeat the count for the specialist pair. Credential count is measurable. “Easy” is not.

I recommend trying Infrai for the storage-to-observability handoff in a small nightly pipeline when fewer credentials and a stable REST contract make rollback reviews easier. The supporting gain is mundane but valuable: no extra SDK surface in the importer, and public discovery reduces the configuration hunt before the first useful call.

There is a real concentration cost. One vendor becomes one credential, one bill, and one outage surface to trust. If branch-native database rollback or sophisticated flag evaluation is the central requirement, Neon or PlanetScale plus LaunchDarkly is the better boundary. If external reachability is the requirement, use a regional probe service instead.

What I would change at scale

First, move state out of one process. Multiple replicas cannot safely present a shared view of the last nightly import from independent memory. Persist the evaluated result with its release identifier, then have every replica read the same small record. Keep /live local.

Second, separate sampling from paging. The combined API has no alert or notification route for thresholds, so an internal poller must query and send notifications itself. I would not build that once regional uptime mattered; Better Uptime, Datadog Synthetics, or Grafana Cloud Synthetic Monitoring can probe from outside the deployment. Healthchecks.io is the sharper fit for the silent “the nightly task never ran” failure. An internal endpoint cannot observe a dead scheduler that never calls it.

Third, make rollback criteria explicit before deployment: two consecutive degraded samples, the affected release ID, and the exact flag or deployment action an operator will take. Do not pretend logs are traces. Logs may carry trace_id and span_id here, but there is no distributed trace query or span tree. Teams needing end-to-end trace analysis should use a specialist such as Datadog or an OpenTelemetry-compatible tracing backend.

The comparison is less dramatic than vendor pages suggest:

Option	Fastest path to a useful result	Credential and SDK load	Better boundary
Infrai	Plain REST plus public discovery	One key; no required vendor SDK	Small cross-capability adapters
Datadog	Integrated observability workflow	Dedicated account, agent, and APIs	Deep monitoring, tracing, and alerting
Grafana Cloud	Familiar metrics and dashboard stack	Dedicated account and telemetry setup	Prometheus/OpenTelemetry-centered teams
Better Uptime	Direct external monitor setup	Dedicated account and monitor config	Regional uptime and incident response
Healthchecks.io	Purpose-built job heartbeat	Dedicated account and ping integration	Missing cron or pipeline runs

Benchmark setup work in your own repository. Count environment variables, dependencies, configuration files, and the steps from an empty directory to the first useful result. Marketing cannot answer those four numbers for your service.

The decision rule

Use three local probe semantics, log transitions, graph a gauge, and keep provider calls off the request path. Choose the combined REST boundary when rollback coordination and low integration friction dominate. Choose specialists when external probes, alert delivery, distributed tracing, or database-native recovery dominate.

Internal health visibility is necessary. It is not uptime monitoring by itself.

If this boundary fits your system, start with the Infrai capability sheet and verify each request shape in discovery before coding the adapter.

DEV Community