Vicente Reyes wrote up a deploy debugging story this week. Ten small breakages stacked in series, each one hiding the next. Good piece. Go read it.
The last act is the one I want to pick up.
Two minutes after his first fully automated deploy, the site returned a 502. Nothing had crashed. The old container was gone, the new one hadn't finished booting, and he happened to refresh inside a window about one second wide.
That window is in almost every Docker Compose deploy I have seen. It is in mine.
The interesting part isn't the window. It's that the obvious fix only closes half of it, and the other half fails much more quietly.
The proxy has no idea
A static Traefik route looks like this:
http:
services:
api:
loadBalancer:
servers:
- url: http://api:3000
That config says: send traffic to api:3000. It does not say: send traffic to api:3000 when something is actually listening there. Traefik has no opinion on the matter. It resolves the name, opens a connection, reports whatever comes back.
So every time docker compose up -d stops the old container and starts the new one, there is a stretch where the proxy is confidently routing at a process that is still running migrations. Nobody hits the app during that stretch, you never find out.
The fix everyone reaches for is an active health check.
http:
services:
api:
loadBalancer:
servers:
- url: http://api:3000
healthCheck:
path: /healthz
hostname: api.example.com
interval: "5s"
timeout: "3s"
Correct, and it's where most write-ups stop.
The hostname line is not optional
Leave hostname out and Traefik sends the check with Host: api, the internal Docker service name. Anything with a host allowlist in front of it will reject that. Django gives you a DisallowedHost 400. Express with a host check does the same. So does NestJS behind most gateway setups.
From Traefik's side a 400 is a failed check, and a container that fails every check is permanently unhealthy.
So you spend an afternoon debugging why your app is down. Your app is fine. The health check is asking the question in a form the app is contractually obliged to refuse.
Set hostname to the real public domain. Then the check looks like real traffic, which is the whole point of it.
Liveness and readiness are different questions
The advice attached to that config is usually "keep /healthz trivial". No database, no auth, just return 200. That advice is right, and for a good reason: if your health endpoint touches a business dependency, a permissions change or one slow query quietly pulls the container out of rotation.
But a trivial endpoint answers a trivial question. It answers is this process alive. It does not answer can this process serve a request.
Those two come apart constantly. A container that boots cleanly but can't reach its database will return 200 from a trivial check all day. Pool empty. Queue consumer never attached. The proxy sees green, sends real traffic, and instead of one second of clean 502s you get sustained 500s from a container that is reporting healthy.
So split them. Liveness stays trivial and wires to your restart policy. Readiness checks what the container needs to do its job and wires to the load balancer. Collapse the two and you pick one failure mode to catch and get blindsided by the other.
The part nobody writes down
This is the bit I actually sat down to write.
The natural readiness check for a service that depends on Postgres is a SELECT 1. Cheap, honest, and it's what I'd reach for too.
Think about what it proves. It proves the pool can hand you a connection. It says nothing about whether the pool is nearly empty.
Under load those diverge, and the divergence is ugly. If readiness borrows from the same pool the application serves from, then when that pool is close to exhausted the check either succeeds by taking the last free connection, which is the moment you least wanted a health probe competing with real requests, or it fails outright because there was nothing left to take.
The second one is what hurts.
A container fails readiness. The load balancer pulls it. Its traffic redistributes to its neighbours. Those neighbours now carry more load, their pools saturate, they fail readiness too, and out they go.
You have built a check whose entire job is to protect the system, and under exactly the conditions it was meant to protect against, it removes the system.
Three cheap ways out, pick one:
Give readiness a dedicated connection outside the application pool. Then a saturated pool is a slow app rather than an unhealthy one.
Or keep it on the shared pool and be honest about what it asserts, which is "Postgres is reachable from this container". Fine thing to check, as long as nobody downstream believes it means "has capacity".
Either way, never let a single failed probe evict. Require consecutive failures over a window so a momentary squeeze can't cascade.
The general principle is that a readiness check consuming the same scarce resource it measures will always give you its worst answer at the worst possible moment.
Readiness is a dependency graph you're signing up for
Everything you put in a readiness check becomes something that can take you out of rotation.
Put a downstream HTTP call in there and you have coupled your availability to somebody else's. They have a bad ten minutes, every one of your containers marks itself unready simultaneously, and now you have zero healthy instances of a service that was working perfectly well.
So the scoping rule is narrow. Check only what this container needs to serve its own traffic.
Vicente landed on exactly this for his stack. Postgres in, Redis and Celery out, on the grounds that background jobs degrading is not the same thing as the web tier being unable to answer. Redis going down should page someone. It should not empty the load balancer.
That's a design decision, not a config line, and it's worth the ten minutes.
Start-first doesn't get you to zero either
Say you fix all of that. The window shrinks from "boot plus migrations plus static assets" down to "process fork time".
It shrinks. It does not close. docker compose up -d stops the old container before the new one is ready, and no health check changes that ordering.
The usual next step is a start-first rollout. Bring up the new container, wait for it to pass readiness, kill the old one. The docker-rollout plugin does this, Swarm and Kubernetes do it natively.
The bit that gets skipped: start-first closes the window where nothing is listening and opens a different one, where the old container takes SIGTERM with requests still in flight.
Two things have to be true for that second window to be harmless. The proxy has to notice the old server left and stop sending it new work before the container dies, and Traefik's config reload is not instant. And the application has to be allowed to finish what it is already holding. In gunicorn that's graceful-timeout. In Node it's whatever you wired into your SIGTERM handler, and if you wired nothing, the answer is that it doesn't.
A rollout without a drain window doesn't eliminate dropped requests. It moves them from the front of the deploy to the back, where they are harder to spot because the deploy reports success.
If your app is anywhere near money that distinction matters a lot. A request dropped at the front is a failed connection and the client retries. A request dropped at the back was accepted, was being processed, and then wasn't.
What to actually do
The answer is different at each size, so:
Single box, Compose, low traffic. Split liveness and readiness, scope readiness to your own datastore, set hostname, live with a health-gated couple of seconds. Completely reasonable place to stop. Reaching for an orchestrator here costs you more than the window does.
Multiple instances behind a proxy. All of the above, plus a failure threshold so one bad probe can't evict, plus readiness on its own connection. This is where the cascade becomes possible, so this is where it's worth paying for.
Deploys that cost you money when they drop requests. Start-first rollout with an explicit drain. Proxy deregistration before SIGTERM, graceful timeout longer than your slowest normal request.
The tools matter less than the question.
Every health check is an assertion, and the failure is always the same one. The assertion is narrower than what everyone downstream believes it to be. Somebody reads a trivial 200 as "can serve". Somebody reads a SELECT 1 as "has capacity". Somebody reads a green deploy as "nobody got a 500".
Write down what your check actually proves. Then go and find out who is relying on it to prove more than that.
Thanks to Vicente G. Reyes for the original write-up and for a good argument in the comments. Most of this is that argument.
I'm Arun, CTO and co-founder at Atoa. We build open banking payments for the UK. I write about the messy parts of running engineering systems. @mickyarun.
Top comments (7)
The composite is the honest part, so I would not try to name it precisely. I would date it.
/healthz-or-whatever-it-is-called settles one thing: "this summary was accurate as of the check that last ran." That sentence is true no matter how many parts you fold in, and it can be kept true by the check itself, not by the person maintaining the repo. The parts list is what degrades, so the parts list has to live where the reader is at 3am, which is the alert body, not the runbook nobody opens.
The part of your reply I keep turning over is the reader. You are right that the rename is a judgement about the reader, and that is why it is not fully in your hands: the person who reads green at 03:14 has never seen the check, the runbook, or this post. They read the name as the answer to the question they already had. Which means the name is not really a name. It is a claim with a population, and the population is whatever the reader assumes it covers. /healthz says "one process is listening" and gets read as "we can take payments". Nobody miscounted. The sentence swapped the population.
That is why I would watch for the moment of renaming-back you named, and treat it as the useful signal it is: not a preference about filenames, but somebody wanting the check to cover a claim it does not make. The question I would put to them is not "why the old name" but "what do you want green to mean when the phone rings". If they can answer that in one sentence, you have the real check and the name follows. If they cannot, no filename saves the incident.
"A claim with a population, and the population is whatever the reader assumes it covers" is the sentence. Nobody miscounted. The sentence swapped the population. I had this filed as a naming problem and it is not one, because a name cannot carry a population at all.
The date works for a reason worth naming, because it is the test for everything else. "Accurate as of the check that last ran" stays true with nobody maintaining it. The check writes the timestamp. There is no edit anyone can forget.
Run that test on the parts list and it fails, including in the alert body. A written parts list drifts the first time somebody adds a dependency to the check and not to the string, and nothing breaks when they do. Same failure class as the check itself. Moving it into the alert body does not fix that, it moves it somewhere more people see it being wrong.
The version that passes your test is a parts list generated from whatever the check iterates, so there is one list and both the check and the message read it. If it has to be typed by hand I would rather it were not there, because a stale parts list is worse than none: it reads as the population and it is out of date.
Where I think the argument stops is the 03:14 reader you put at the centre of it. The parts list in the alert body is not read at 03:14 either. They read the colour. The timestamp survives that and the parts list does not, and not because of where it lives. A stale date is visible without parsing anything. A missing dependency is not.
The parts list is where I was wrong, and you found the reason: I moved it, I did not generate it.
Anything a human types into the message drifts, and a stale population is worse than no population, because it reads as the answer. One object, iterated by the check and rendered into the message, is the only version of that list I would defend. If it has to be typed by hand, leave it out.
What I would keep is a smaller claim than the one you are testing. Not "the list is accurate" but "the list is the thing that ran". A generated list cannot be more complete than the check, which is fine. It cannot be wrong about what the check did, which is the point. Same shape as your timestamp: the property comes from the fact that nobody writes it.
And on the 03:14 reader, you are right and I was arguing the wrong defence. They read the colour. No list in any body reaches them. That is why the rename is load-bearing and the parts list is not borrowed by them at all. The reader who gets the list is the one at 09:00 opening the incident, and their question is not "what is this" but "what did it cover". Different reader, different artefact. My mistake was dressing the 09:00 list up as the 03:14 fix.
Your smaller claim is better than the one I was testing, and I'll take it: the list can't be wrong about what the check did, and that's a provenance claim, not a coverage one. Provenance is the only thing a check's own body can honestly carry.
Where it still lets someone through: a generated list is unfalsifiable in the direction that hurts. The check iterates three of seven dependencies, the list prints three, every word of it is true, and nothing in the message knows seven was the number. Correct about what ran, silent about what didn't. So the generated list buys you "this is real" and not "this is all" — and the second one is what the 03:14 reader would want, if they were reading.
Which puts a count from somewhere else back into the message: reached against intended, where intended comes from the service registry or the dependency manifest rather than from the check. Same shape as your timestamp, one layer out. The denominator has to be written by something that isn't the check.
And yes on 03:14. That's the argument for the rename being the whole fix rather than part of it. If colour and name are the only two fields that reach a person mid-incident, every other field in that body is written for the post-mortem on Tuesday morning. Worth saying out loud, because most health check bodies are written as though someone reads them while paged.
You are right that the denominator cannot come from the check. I think the fraction you build out of it fails the same test one layer further out.
3 of 7 declared, rendered as a number, is a ratio, and a ratio invites a threshold. Whoever decides that four unexercised is amber and three is red has made a policy call inside an artefact that is supposed to be a measurement. Same coat as an invented cost line: a number the reader treats as measured because it printed.
What I would keep from your move is the direction of falsifiability. Your numerator is checkable against the check's own body. If the message says three and the code iterated four, someone can catch it. Your denominator is checkable against nothing except the manifest it came from, and the manifest describes what was declared, not what was required. So the ratio is reached over declared, and the sentence a reader hears is reached over required. A wrong numerator is a bug. A wrong denominator is a belief.
Which is why I would not render the fraction at all. Render the gap as its own field, and make it non-aggregating: "four declared dependencies not exercised by this run". That is either zero or it is not, it can never turn green, and it needs no threshold to be read. A count of absences survives drift better than a proportion of them, because nothing rounds a gap up.
The
hostnamehalf is the one I am stealing.There is a version of this that is not only about Traefik. A health check is a claim, and a claim should name what it settles.
/healthzreturning 200 settles "a process is accepting connections." It does not settle "this service can answer." When those two come apart, the checker keeps reporting green, and the green is accurate, about the wrong question. Same shape as an audit log that agrees with itself.Which is why the split that matters is not trivial vs real. It is: what does this line actually settle, and who is reading it as the answer to something else.
The rename is the part I would act on.
My split was trivial vs real, which is a judgement about the check. Yours is about the reader, and that is the one that breaks. Nobody misreads a check because it is shallow. They misread it because the name promises more than the body delivers.
/healthzsettles "a process is accepting connections". The failure is not the endpoint. It is the incident call where someone reads green as "we can take payments". Both statements are true, about different things, and the minutes go into working out which question the green had answered.So the fix I did not write: put the claim in the name.
/accepting-connectionsis ugly and it cannot be misread. And the moment somebody wants to rename it back to/healthz, you have found the gap — they want it to settle something it does not.Where it stops working for me: some checks genuinely settle several things, and naming those honestly gets you
/db-and-queue-and-issuer-reachable, which nobody keeps current. I do not have a better answer there than keeping the composite with its parts listed, and treating it as a summary rather than a claim.And yes, same shape as the audit log. The log agrees with itself, the check answers itself. Both are internally honest, which is exactly why neither one trips.