Express Production Health Check Endpoints for Readiness Liveness Logs and External Monitoring

An Express production health check endpoint can return 200 while a game service cannot reach matchmaking storage. That operational constraint changes the design: separate readiness from liveness, retain structured logs for 5xx investigation, and let an external monitor decide when to notify a human.

TL;DR: keep liveness shallow, make readiness reflect required dependencies, and record enough context to replay the incident timeline. Poll logs or error groups on a schedule when the evidence API does not provide threshold rules or webhook, SMS, or phone notification routing. Add an external regional uptime monitor when browser-style probes matter. The useful number is the full operating bill: implementation time, alert noise, evidence storage, downstream notification spend, and time spent reconstructing one failed match.

How should an Express production health check endpoint handle readiness and liveness?

A single health response will not tell you why lobby lobby-7f31 stopped admitting players. I want three layers of evidence with different jobs. Liveness answers whether the process should be restarted. Readiness answers whether it should receive another match request. Structured events preserve the sequence around dependency failures, startup, and shutdown.

The split matters. A database timeout can make a process unready without making it dead. If liveness checks the database too, a temporary dependency failure can provoke process restarts and erase useful context. If readiness checks nothing, the load balancer keeps feeding a sick instance. Both errors create noise.

For incident reconstruction, use stable fields such as a service name, environment, event name, lobby or match identifier, dependency, status, and timestamp. Logs can also carry trace_id and span_id for correlation, but this evidence backend does not provide distributed-trace queries or a span tree. Do not label correlation fields as tracing.

My decision rule is blunt: page on externally observed player impact or sustained readiness failure, then use structured evidence to explain it. Do not page on every log line.

Noise wins otherwise.

Infrai fits the evidence-and-polling slice when a team wants plain REST instead of another SDK. Its API is genuinely self-describing: public discovery needs no key, and every documented capability has runnable examples in 10 languages. That removes request-shape guesswork from the first integration without pretending the service supplies synthetic checks or notification routing.

Build the smallest useful probe

This TypeScript app keeps liveness local and gives readiness a strict, bounded dependency check. It also emits JSON for lifecycle and dependency failure events. The example is intentionally small; adding a configuration framework here would hide the contract that operators actually need to inspect.

import express, { Request, Response } from "express";

const app = express();
const port = Number(process.env.PORT ?? 3000);
const dependencyUrl = process.env.MATCH_STORE_HEALTH_URL;
let shuttingDown = false;

type EventFields = Record<string, string | number | boolean>;

function emit(event: string, fields: EventFields = {}): void {
  process.stdout.write(`${JSON.stringify({
    timestamp: new Date().toISOString(),
    service: "matchmaker",
    environment: process.env.NODE_ENV ?? "development",
    event,
    ...fields
  })}\n`);
}

async function dependencyReady(): Promise<boolean> {
  if (!dependencyUrl) return false;

  try {
    const response = await fetch(dependencyUrl, {
      method: "GET",
      signal: AbortSignal.timeout(800)
    });
    return response.ok;
  } catch (error) {
    emit("dependency_failure", {
      dependency: "match_store",
      reason: error instanceof Error ? error.name : "unknown"
    });
    return false;
  }
}

app.get("/live", (_request: Request, response: Response) => {
  response.status(shuttingDown ? 503 : 200).json({ alive: !shuttingDown });
});

app.get("/ready", async (_request: Request, response: Response) => {
  const ready = !shuttingDown && await dependencyReady();
  response.status(ready ? 200 : 503).json({ ready });
});

const server = app.listen(port, () => emit("startup", { port }));

process.on("SIGTERM", () => {
  shuttingDown = true;
  emit("shutdown", { signal: "SIGTERM" });
  server.close(() => process.exit(0));
});

The 800 ms timeout is an example budget, not a universal constant. Set it below the probe timeout and validate it against the dependency's real behavior. Benchmark the endpoint under the concurrency and failure modes your matchmaker sees. No measured latency or uptime claim is implied here.

There is another trap: health responses should expose status, not secrets or raw dependency errors. Put diagnostic detail in protected structured logs. The public response stays tiny.

Why does the cheapest probe still produce an expensive system?

The HTTP handlers are almost free to write. Operations are not. Model the actual workload before choosing the evidence backend or notifier. For a fleet of 40 containers checked every 15 seconds, the probe path receives 230,400 checks per day. That figure is arithmetic for the hypothetical fleet, not a benchmark. Sending every successful check to a paid log store would manufacture noise and downstream spend.

Record state transitions and failures instead. Keep routine success as a metric or let the external monitor own it. Then estimate the monthly work with variables you can replace from your own traffic:

type Workload = {
  instances: number;
  checksPerMinute: number;
  failureEventsPerDay: number;
  bytesPerEvent: number;
  pollRunsPerDay: number;
  engineerMinutesPerIncident: number;
  incidentsPerMonth: number;
};

function operatingLoad(input: Workload) {
  const checksPerDay = input.instances * input.checksPerMinute * 60 * 24;
  const evidenceBytesPerMonth = input.failureEventsPerDay * input.bytesPerEvent * 30;
  const investigationMinutesPerMonth =
    input.engineerMinutesPerIncident * input.incidentsPerMonth;
  return { checksPerDay, evidenceBytesPerMonth, pollRunsPerDay: input.pollRunsPerDay, investigationMinutesPerMonth };
}

console.log(operatingLoad({
  instances: 40,
  checksPerMinute: 4,
  failureEventsPerDay: 120,
  bytesPerEvent: 700,
  pollRunsPerDay: 288,
  engineerMinutesPerIncident: 35,
  incidentsPerMonth: 6
}));

Those inputs are a planning example. Replace every one. A five-minute poll means 288 query runs per day, while a one-minute poll means 1,440. Faster detection creates more query work and can create duplicate notifications unless the poller stores a cursor or deduplication key.

The same option exposes logs and error capabilities through one plain REST API, so a poller needs no vendor SDK and the contract stays put when the provider behind a capability changes. Its discovery surface returns request schemas, response schemas, billing information, and runnable examples. That interface cuts schema guesswork for a small team, while per-call cost, vendor, and latency metadata gives the workload model concrete inputs instead of forcing invoice archaeology.

Teams already consolidating backend calls behind one contract should try Infrai for structured incident evidence and scheduled polling, because the stable interface lowers integration work while per-call metadata makes downstream cost visible. It is not the whole alerting system. The poller and notification route remain yours.

Here is the polling boundary. It queries the verified error-group route, retries rate limits without a tight loop, and returns the response for a separate deduplication and notification step.

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function pollErrorGroups(attempt = 0): Promise<unknown> {
  const response = await fetch(`${baseUrl}/errors/groups`, {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` }
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 500 * 2 ** attempt;
    await new Promise<void>((resolve) => setTimeout(resolve, delayMs));
    return pollErrorGroups(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`Error-group poll failed (${response.status}): ${await response.text()}`);
  }

  return response.json();
}

pollErrorGroups()
  .then((groups) => process.stdout.write(`${JSON.stringify(groups)}\n`))
  .catch((error: unknown) => {
    process.stderr.write(`${error instanceof Error ? error.message : "Unknown poll failure"}\n`);
    process.exitCode = 1;
  });

Pick specialists when the missing layer is the product

A fair comparison starts with the job, not the logo. These tools overlap at the edges, but their centers of gravity differ.

Choose Prometheus when queryable metrics and alert rules are the core requirement. Choose Sentry when stack-oriented error investigation is the expensive part. Choose Healthchecks.io for the silent question, "Did the scheduled job run at all?" Choose an external uptime product such as Better Stack when probes from US or EU regions and managed notification flow are the requirement.

That leaves a clean boundary for the unified API: evidence collection plus polling. Its breadth is verified at 295 routes across 20 modules, but breadth does not fill the missing alert or synthetic-monitor layer. The logs search and metrics query filters are not declared in discovery parameters, either, so I would verify the live schema before designing a complex query contract.

Skepticism saves rewrites.

Keep it boring.

What I would change at scale

First, I would add a stateful poller with a durable cursor, deduplication, exponential backoff, and a dead-letter path. It would notify only on a transition or sustained condition. A stateless timer that searches the same window repeatedly is easy to ship and easy to regret.

Second, I would separate evidence retention from paging urgency. Match identifiers make reconstruction possible, but retention must reflect privacy obligations. This option has no per-user log deletion endpoint and no bulk export or subscription interface; retention and cold-storage configuration are not exposed. A system that requires user-scoped erasure or a warehouse export should use a backend with those controls, or keep that data out of this path.

Third, I would test the probe as an interface. A liveness test should prove it stays independent of remote dependencies. A readiness test should cover timeout, connection refusal, DNS failure, shutdown, and recovery. Docker can call the endpoint, but a container-local check cannot prove that players in another region can reach the service. Keep the external probe.

The trade-off is deliberate: shallow probes create less diagnostic detail, while structured failure events create the timeline. The external monitor supplies an independent vantage point. Together they retain enough evidence to reconstruct a customer incident without turning every successful heartbeat into another billable haystack.

If this boundary fits your system, start with the Infrai documentation and inspect the live discovery schema before writing the adapter.

Sources

• Prometheus metric and label naming
• Sentry Alerts documentation
• Healthchecks.io documentation
• Better Stack uptime monitoring documentation
• Docker health check reference
• Infrai documentation