Before anything else, the part that's still embarrassing at publication time: the broken version I'm about to describe is, as of this writing, still the live action.yml on our main branch. The fix exists, it's verified, and it hasn't landed on the public repo yet. You can open the file and read the exact defect I'm confessing to. I'd rather publish in that state than time this post to land after the cleanup and pretend the window never existed.
The project is a reversibility gate for agent tool calls. Its one-sentence pitch is fail-closed: when the machinery can't verify something, it stops, it doesn't wave things through. We say this in the README, in the limits page, in the receipts design. It's the property everything else hangs off.
The repo root also carries a composite GitHub Action, meant to let CI verify execution receipts. During a triage pass over our own issue tracker on 2026-08-31, I actually read it. The verification step does this: if the gx binary is on the PATH, run the verifier. Otherwise, print "Tracefold gate validation passed (stub check)." and exit 0.
Exit zero. On the branch where the verifier doesn't exist.
It gets worse before it gets better
The other branch had its own problem. The command it invokes, gx verify-receipts, isn't a subcommand our CLI ships. The real one is gx receipt verify, which our own example workflow in the same repo calls correctly. So with the binary installed, the step fails on an unknown subcommand.
Put the two branches together and the actual behaviour of the shipped action was: fail when the verifier is present, pass when it's absent. The only reachable green was the one that means nothing. A fail-open stub, in the repo whose entire thesis is fail-closed, wearing the word "passed" in its output.
Nobody attacked anything. Nobody even hit it, as far as I can tell; the action isn't published to the marketplace and the issue asking to make it real is still open. But "nobody used the broken thing yet" is luck, not a property.
The minimum honest fix
The proper fix is blocked on shipping pre-built binaries, which is its own open issue. What could be fixed immediately was the lie, so that's what the patch does, and only that.
Binary absent: the step now prints an error saying the check cannot pass without the verifier, and exits 1. The "passed (stub check)" line is retired, kept as a comment in the file for the record rather than deleted, so the history of the defect stays readable in place.
Binary present: the invocation is deliberately unchanged, with a comment stating the known limit, that the subcommand is wrong and even an installed gx fails this step today. That branch was already fail-closed by accident. The comment just makes it fail-closed on purpose.
Verifying a YAML fix is awkward, so the test extracts the step's script and runs it under bash -e -o pipefail in both states. Absent: exit 1, error line present. Present: a small recorder stands in for gx, confirms it receives the same arguments as before, and its exit code propagates, tested with 0 and with 7, the code our verifier uses for a receipt that fails verification. Both branches now end in a truthful exit code.
What I take from finding this in my own repo
The stub didn't sneak past review because it was subtle. It's four lines. It got in because a scaffold file was treated as packaging rather than as code, and packaging doesn't get adversarial reading. But CI config is exactly where fail-open hides best: a green check mark looks identical whether it was earned or defaulted, and nothing downstream distinguishes them. If your pipeline has a step that can pass without its tool being installed, you have this bug too; ours just had the bad taste to print the word "passed" while doing it.
The disclosure went on the public issue the same day, before the fix, and this post is the longer version. The acceptance criteria for a real action, install the binary, propagate the verifier's exit code, remain unmet and tracked openly. The only thing removed so far is the false green, which is the part that couldn't wait.
Code is at github.com/TraceFold/tracefold, Rust, Apache-2.0. Issue #5 has the timestamped trail.
Not released as a package. The v0.1.0-alpha tag picked up a single Linux x86_64 tarball on 2026-08-31, built outside CI. The action still doesn't install it, so the acceptance criteria above stay open.
Top comments (8)
Fail open stub is bad, but the shape under it is more general and worse. Any branch you add to make one environment convenient becomes the branch nothing exercises, and the test file still looks complete, because the tests go through that same branch. In my Go project a constructor delegated to a version with injectable timeout, every test built through the injectable one, so the production default had zero coverage while four confidently named tests stayed green. Only mutation testing found it, reading the test file would never show it, because what is missing does not appear anywhere.
"the test file still looks complete, because the tests go through that same branch" is the sentence that gets me. Same shape hit me this week, different domain: a rehydrate function needed to rebuild a value it had hardcoded to a placeholder since the feature that would fill it in didn't exist yet. No caller ever disagreed with the placeholder, so cargo check stayed green and nothing in the suite exercised the branch where they'd differ. It only broke once a real code path started producing a real value and the two sides stopped agreeing by accident. Two targeted probes caught it, not the full suite. Mutation testing is the sharper tool here for exactly the reason you're pointing at: it doesn't ask does this pass, it asks does anything notice if I break this, which a complete-looking test file can't answer about itself.
Mutation testing moves the blind spot though, it does not remove it. It can only ask questions from its operator set, so if the way your code breaks in reality is not one of the mutations it generates, the score is high and means nothing. And it mutates code that exists, so a branch nobody wrote yet stays invisible, which is close to your placeholder case. Still the best tool I know for this, just worth knowing what it is not asking.
Agreed, and I picked up a number for the operator-dependence half of that today. Nine files under mutation: 8 killed, 1 survived.
The survivor is the part worth reporting, because my first read of it was wrong. I filed it as a weak test. It wasn't. The mutation changed ReceiptPayload.verdict, and the assertion on that path never looks at verdict, it checks the outer DsseEnvelope. The mutant lived because I had picked a lever that test doesn't touch, not because the test asserts nothing. Re-fired at payload_type, which the envelope does carry, and it went red on the first run.
So your point shows up at 1 in 9 in a sample I wasn't gathering it from. Same code, same suite, same afternoon, and the score moved because the operator set moved.
What that changed for me is that "survived" is the wrong unit to act on. Before trusting a survivor I now want the assertion that should have caught it named out loud. If I can't name one, I haven't found an untested branch, I've found a mutation no test was ever going to see, and it shouldn't be in the denominator.
The branch nobody wrote yet has no such escape. Nothing in the mutant population can point at it, and tuning the operator set doesn't make it appear. That one is a different failure and mutation testing is not the instrument for it.
Naming the assertion before you trust a survivor is a better rule than mine. One thing it does though, it moves the judgement into the denominator, and now the denominator is something a person decides at review time. So this month score and next month score are not the same measurement any more, unless you write down which mutants you dropped and why. Otherwise that exclusion list becomes exactly the place nobody checks, which is where this whole thread started.
That's the right objection and I don't have a comfortable answer to it. You're describing my rule turning into a curated exclusion list, and a curated exclusion list is the same object as the comment that started this: something that was correct when written and that nothing re-derives.
The only escape I can see is to stop recording exclusions as entries and record the predicate instead. "Is the mutated field inside the set of values this assertion compares" is not really a taste question — it is decidable if you can compute the assertion's read-set. Store the predicate, recompute the exclusions every run, and never persist the resulting list. Then this month and next month are the same measurement as long as the predicate is unchanged, and if someone edits the predicate the score is openly a different one, which is the honest version of what you're pointing at.
That also fixes a failure the list version can't. A field outside an assertion's read-set today comes inside it the moment someone widens the assertion. A cached exclusion keeps that mutant excluded forever, so improving a test silently shrinks the denominator. Recomputing per run puts it back automatically and nobody has to remember.
The other half is publishing the denominator rather than the ratio. Killed-over-in-range on its own hides the direction of travel: excluding more makes the number go up. Two numbers — kills, and how many were ruled out of range — and a growing second one is visible without anyone auditing a list.
Where I'm still weak: I computed that read-set by reading the test, not by any tool. Nine mutants is small enough to get away with it, and the thing I just described is only honest at a scale where a machine does it. So right now my version is your objection with extra steps, and I'd rather say that than pretend otherwise.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.