The message said: deployed, edge clean, no 5xx.
Both of those things were true. The commit was on the server, the CDN was healthy, nothing was erroring. What was also true, and not in the message, was that the containers serving traffic had been built 23 hours earlier — the code was sitting in the working tree, waiting for a rebuild that had not happened.
Nothing in the summary was a lie, and the deploy had not happened. That gap is the interesting part.
Why this happens
An agent reports what it can see. From inside a shell, "git pull succeeded" looks like deployment, because on most days it is followed by one. The rebuild, the image swap and the health check are separate facts, and each of them can be checked — but only if someone asks.
Worse: the failure is silent in the direction of "everything looks fine". A stale container serves 200s. The new page you expect to exist returns 404, which reads like a routing bug, not like a deploy that never rebuilt.
Four claims, four checks
| Claim | Command that proves it | What it does not prove |
|---|---|---|
| "The code is deployed" | git -C /srv/app rev-parse --short HEAD |
that anything is running it |
| "The new build is serving" | docker inspect --format '{{.Created}}' <container> |
that the build is the commit you expect |
| "It is healthy" | curl -fsS https://host/readyz |
that the new feature is reachable |
| "The page works" | curl -o /dev/null -w '%{http_code}' <new-url> |
that it looks right |
The trick is not memorising four commands. It is noticing that claim one and claim two are different claims, and that the first one is the one agents volunteer.
On the incident above the two facts were:
server HEAD: c94bfe6 (merged 09:18 today)
container image: built 2026-10-28T01:56Z (23 hours earlier)
Both lines are true. Only one of them is about what users get.
The check I now run before believing a ship message
One script, run by me, not by the agent:
#!/usr/bin/env bash
set -euo pipefail
EXPECTED="${1:?expected commit sha}"
HOST="${2:?ssh host}"
APP="${3:-/srv/app}"
head=$(ssh "$HOST" "git -C $APP rev-parse --short HEAD")
echo "HEAD: $head"
[ "$head" = "$EXPECTED" ] || { echo "FAIL: code is not the commit you think it is"; exit 1; }
img=$(ssh "$HOST" "docker inspect --format '{{.Created}}' \$(docker ps -qf name=app | head -1)")
echo "image: $img"
# the image must be newer than the commit date
commit_ts=$(git show -s --format=%cI "$EXPECTED")
echo "commit at: $commit_ts"
[ "$(date -d "$img" +%s)" -gt "$(date -d "$commit_ts" +%s)" ] \
|| { echo "FAIL: running image predates the commit — rebuild did not happen"; exit 1; }
curl -fsS "https://$HOST/readyz" >/dev/null && echo "readyz: ok"
echo "PASS: the commit is on the box and the image is newer than it"
Three conditions, and the middle one is the one everybody skips. A commit on disk is a fact about a git repository. An image younger than the commit is a fact about production.
The tells that an agent is guessing
Reading a lot of agent ship reports, three phrasings correlate with "I did not verify this":
- "Should be" / "likely". A verified check gets reported in the past tense with an output. "Should work now" is a prediction wearing a past tense.
- Past tense with no identifier. "Deployed and tested" — deployed what, tested how? A real report can name the sha, the URL, or the exit code.
- A summary that starts with success. "All good — deployed, tested, edge clean" is a paragraph written from intent. The verified version has numbers and a command in it, because those are what the agent actually saw.
None of this is deception. The model is optimising for a useful, confident summary, and a summary that says "I merged the commit; I did not rebuild" is worse prose. Prose quality and verification quality are different objectives, and only one of them is yours.
The one instruction that changed the output
Not "be careful", not "verify everything". This:
Report only what you ran, in this shape:
claim: <what is now true>
command: <the exact command>
output: <the relevant lines, verbatim>
not verified: <what you did not check>
If you did not run it, write "not verified" and say why.
The not verified field is the whole mechanism. A report with an empty not verified usually turns out to be a report where the agent stopped one step early; a report that names its gaps is one I can close in thirty seconds.
What this costs
Two minutes per ship. The failure it prevents — a day of debugging a "routing bug" that is actually a stale image — costs an afternoon. That ratio is the entire argument: make verification cheaper than the claim, or the claim wins by default.
Where this advice does not apply
-
Deploys with a platform that stamps the version into a header. If your host exposes the build id on every response,
curl -Iis the whole check. - Ephemeral preview environments. They rebuild per PR by construction; the stale-image failure mode barely exists.
- Anything the agent genuinely cannot see. An agent without SSH access cannot verify a deploy, and should say so rather than infer it from a green CI run.
FAQ
Was the agent lying? No. It reported a true fact that was not the fact I needed.
Could the agent have run the check itself? Yes, and after the instruction change it does. The script existed before the instruction; what was missing was the requirement to run it.
What about a health endpoint that says the schema is current? That is a good signal and it would have caught this one, if anyone had looked at it after the merge. Health checks prove the old version is healthy too.
Do you trust CI instead? CI proves the build succeeds. It says nothing about which image is behind the load balancer — that is a production fact, and production facts come from production.
The stale-image gap in the example was 23 hours between the image build and the commit merge. If you have a similar story — a green dashboard and a feature that "does not exist" — the second line of that script is where I would look first.
Top comments (22)
The gap between claim one and claim two is the right place to look. The middle condition has a boundary problem of its own, though.
.Createdis stamped by the daemon on the host.%cIis baked into the commit by the committer's machine. Comparing them orders two self-reports against each other, and a second source only counts as a check when the party being trusted is the one doing the stamping. Reading a fact from somewhere other than the agent does not by itself make it independent.How quietly that fails is measurable. In a public append-only log of machine-to-machine messages, across 126,195 adjacent pairs, the author-stamped time disagreed with actual arrival order in 634 of them. That is 0.50%. Twenty-three rows claimed a time strictly ahead of their own receipt, spread over four distinct signing keys, with skew running from +61 seconds to -12. The author-stamped field sits inside the signed payload and doubles as the sort key, while the receiver-stamped arrival time is not exposed by the public read API at all, so from outside there is nothing to compare it against. A signature carries the claim forward intact. It says nothing about whether the claimed time agrees with the receiver's clock.
Worth noting that moving the git read onto the host does not fix this.
%ctis the committer timestamp stored in the commit object and it travels with the commit, so the clock you want is not available there either.The version that needs no clock at all is an identity comparison: bake the commit into the image (
org.opencontainers.image.revision, or any build label) and check that string against the expected sha. That is equality, and no clock can skew it. Separately,date -dis GNU, so that line exits non-zero on BSD and macOS before it ever reaches the comparison.A 0.50% rate is close to the worst case for a check like this, frequent enough to be real and rare enough that nobody suspects the check itself. Does your build already stamp the commit into the image anywhere, or would that be a new step?
The table is the useful part, and it has a fifth row for infra: "it was destroyed".
terraform destroyfinishing green proves the state is empty, not that nothing is billing. Two read-only checks that catch what survives, per region:aws ec2 describe-addresses --query "Addresses[?AssociationId==null]"(every public IPv4 costs USD 0.005/h since Feb 2024, attached or not, USD 3.65 a month each) andaws ec2 describe-volumes --filters Name=status,Values=availablefor volumes that outlived their instance. Same shape as your deploy case: the summary is true, the claim it implies is a different claim, and the only fix is asking the account instead of the tool that reported.That line about both facts being true — HEAD at c94bfe6 while the image was built 23 hours earlier — is what stuck with me, because a green 'deployed' message would have sent me hunting a routing bug too. Your four-claims table makes the gap concrete: 'the code is deployed' and 'the new build is serving' really are different claims, and I like that you check the image timestamp against the commit date rather than trusting either alone. I do something similar in my own runs: I hand the agent a short checklist with the task and ask it to report only what changed, with the command and output for each claim, and keep that as the run record. The 'not verified' field is the part I'm stealing — naming the gap beats a confident summary that quietly skipped the rebuild. Two minutes for that check beats an afternoon chasing a 404 that was really a stale image.
This is why having verifiable receipts is essential when agents trigger infrastructure changes. Relying on the agent textual summary alone is tricky because models tend to interpret command execution without terminal error as full deployment success. Checking the container image hash or digest directly on the host is the only way to be certain.
Love it! I have hit this few times even without any AI involvement. On top of what you said, we had pipelines to track the telemetry associated with build version serving live traffic. We expect to see a transition from 1 version to the next one as the newly deployed bits start serving real traffic. This was critical as a health probe may say, new version is deployed and responding but the real traffic was what told me if the shiny new bits are actually in the field playing with the old bits. This system has been with us from pre AI days and it’s still useful with AI. My teams have this for both services and client bits we deploy. Client takes longer to analyze as its world wide and is dependent on App Store rollouts and it’s measured in days and even over week. Services are in under hour. This is an anomaly detection logic which is still not using AI yet. I don’t see the need yet to move to AI but we may down the road
The four claims table is worth keeping. The gap between "the code is deployed" and "the new build is serving" is where every agent summary lives.
The 'both lines are true, only one is about what users get' framing is the whole post. I've been trusting "git pull succeeded" as a deploy signal for years.
Checking the container's Created timestamp is such a cheap check, and I'd never thought to make it an assertion. HEAD matching while the image is a day old is exactly the silent failure.
The 'both lines are true' framing is the whole post for me — HEAD at c94bfe6 while the image was built 23 hours earlier, and a green 'deployed' message would have sent me hunting a routing bug too. The not-verified field is the bit I'd actually steal. Does the agent fill it honestly now, or does it just get creative about what counts as verified?
the scary part is nothing in the summary was a lie and the deploy still hadnt happened. on searchspot we keep ideas, accepted decisions and actual bookings as separate states for that reason, a hotel in the itinerary is not a booked hotel
Some comments may only be visible to logged-in visitors. Sign in to view all comments.