A deploy platform's reputation comes from its worst failure. Not its feature list. The worst failures are quiet ones: a health check that passes on a container that already died, a certificate that fails to renew at 3 a.m., a rollback target that got garbage collected.
Part 7 of 7 in the Deploy and operate Levelrail series: 1. Install a Self-Hosted PaaS on Your Own Server in Five Minutes; 2. Roll Back a Failed Docker Deploy Without Downtime; 3. Restart a Docker Container Without Downtime; 4. Node Graceful Shutdown in Docker: Handle SIGTERM on Deploy; 5. Update the PHP Docker Image Security Patch, No Downtime; 6. Docker Container Restarting in a Loop: 5 Causes, Fixed; 7. Why a Failed Health Check Still Went Live (this post).
We build Levelrail, a self-hosted deployment platform. During a local dogfood run on 2026-09-27 we found one of those quiet failures in our own code. This is the writeup.
What we saw
The test was simple. Push a commit whose health route returns a 500, and check that the platform rejects the deploy while the old version keeps serving.
Readiness failed after 60 seconds with a ReadinessFailed condition. Up to there, correct. Then the surprise: the ingress routed traffic to the new, failing container anyway. The previous container kept running, unrouted. The platform recorded the deploy attempt as succeeded.
In other words, the system knew the container was unhealthy and sent users to it.
Why it happened
Two separate pieces each did something reasonable on their own.
The application controller tracked readiness correctly and wrote a Ready condition. The ingress, in a different part of the system, picked which container to route to by a simpler rule: the container named after the desired image, as soon as it ran. Running, not ready.
Nothing connected the two. The readiness verdict existed, and the router never read it. And the deploy attempt record got written when the container started, so it never changed to failed when readiness later failed.
Lesson: two components that each hold half of a safety guarantee do not add up to a guarantee. Someone has to own the whole path from "probe failed" to "no traffic".
The fix
Two changes, shipped in a single pull request:
- The routing step now gates on the application controller's stored Ready condition. If the new release is not ready, it falls back to the previous release that is still running, rather than cutting over early.
- A failed readiness probe now flips the deploy attempt to
failed. The history shows what really happened.
We also added tests for both: one that fails readiness and checks the routing choice, one that checks a fresh deploy with failing readiness records the failure, and one at the storage layer that checks the newest succeeded attempt flips to failed.
Related bugs found in the same run
The same run turned up nearby problems. A rollback to a mistyped image tag returned exit code 0. The container could not start, the ingress dropped the route, and the app went offline while the old container was still running. The CLI now refuses a tag that is not present locally, names the tags it does know, and points at the right command.
A www redirect also disappeared after an env import, because every app save deleted and re-inserted the domain rows that redirect and certificate settings hang off. We fixed that one too.
None of these show up in a feature list. They showed up because we ran the platform against a real app and tried to make it fail on purpose.
What the platform checks now
Beyond routing on readiness, the controller verifies that the running container's image matches the pinned digest the deploy intended. If it does not, the app reports Ready: False with ImageDigestMismatch and the previous release keeps serving. It never reports Deployed while the running content differs from what you shipped.
You can check a deploy's outcome from the CLI, and the exit code tells you the result:
levelrail-cli apps deploys wait web
Zero means healthy. Seven means failed, canceled, superseded or blocked. Six means the wait timed out.
What is still open
We are not done. One item from that run was open when we wrote this: confirming end to end that apps deploys wait reports a readiness failure correctly now that the attempt flips to failed. We expect it works as a side effect of the fix, and we have not verified it independently.
Levelrail is young. The project labels single-server deployments production ready, and newer features are still beta. Run it beside your current setup for a full business cycle with real traffic. That is what surfaces the boring failures.
Go deeper
- Deploy safety: digests, stale guards and freeze windows
- Deploy failures and what each code means
- New to Levelrail? Start with the install guide.
Levelrail is open source under Apache 2.0. Bugs and requests go in the issue tracker.
Frequently asked questions
What is the difference between a running container and a ready container?
Running means the process started. Ready means its health or readiness probe passed. Routing traffic on running instead of ready sends users to a container that may not work.
What do the exit codes of levelrail-cli apps deploys wait mean?
Zero means the deploy is healthy. Seven means it failed, got canceled, superseded or blocked. Six means the wait timed out.
What happens if the running image does not match the deployed digest?
The app reports Ready: False with ImageDigestMismatch and the previous release keeps serving. It never reports Deployed while the running content differs from what you shipped.
Previous: Docker Container Restarting in a Loop: 5 Causes, Fixed
All parts: Deploy and operate Levelrail
Originally published at https://levelrail.com/blog/health-check-passes-dead-container/
Top comments (3)
The fix is right from the read side. There's a second false green left on the wait side.
I read main at 00107a9.
computeDeployOutcomechecksstore.DeployAttemptStatusFailedbeforeRolloutStateServing, so the flip does reach the poller: a readiness failure that marks the attempt failed comes back asfailed,apps deploys waitstops polling and exits 7. The ordering is the load-bearing part. If serving were checked first, the same attempt would still read healthy after its probe failed, and the CLI would be back to reporting success on a deploy the platform knows is broken.What I do not see covered is the implicit deploy id.
WaitForDeployresolves"latest"on the first poll and then pins that id for the whole wait, and the server resolveslatesttoattempts[0]at request time. Soapps deploys wait web, started before the new attempt row exists, follows the previous deploy, which is already terminal, and exits 0. Same shape as the bug in the post: the wait is reading "the newest deploy I could see is healthy" when what it means is "some deploy is healthy".You already guard one case of this: newest deploy healthy while a freeze is active exits blocked instead of reporting the old release. That works because the freeze explains the missing attempt. A deploy that never became an attempt row (a webhook that died before
beginBuildDeployAttempt, a quota rejection, a trigger that errored) has no explanation, and the wait hands back the previous release's 0.For the end-to-end check you list as open, that window is where a false green would show up: push the commit whose health route returns 500, run the wait, get 0 from the release that was already serving.
The guard I would add, local to the wait: keep the id it resolved and the time the wait started. If the resolved attempt finished before the wait started, and no newer attempt appears within the first poll or two, exit non-zero with its own code instead of 0. Explicit id stays the only way to wait on an older deploy on purpose, which is what your freeze note already tells people to do.
On the part you expect works as a side effect: I think you are right, and because of that ordering, not because the flip and the wait were designed together.
One question about the E2E run: do you start the wait after the trigger returns a deploy id, or from the push? If it is from the push, the race is inside the test.
Two components each holding half a correct system is the scariest failure shape — no bug to point at, just an edge that never got drawn. The verdict existing and the verdict being consumed are different features. The test that catches the class is cheap: ship a build that returns 500 on the health route on purpose, then assert the old container still owns the traffic. Everyone tests that green deploys; almost nobody tests that red does not. One deliberate failure in CI buys the whole category, and it stays green-quiet until the day it matters.
routing on running instead of ready is the quiet failure.