I spent a day building a tool whose entire premise is that you should not trust a system's own report of itself. Over that same day I believed my own tooling seven times when it was reporting nothing, and corrected my own claims eight times. Here are the four that were worth the price.
The check that compared two files that were not there
A gate compared a generated artifact against its committed copy and passed when the bytes matched. Both paths had moved. It read two missing files, got two empty strings, found them equal, and reported green. It had been green for a while.
Byte-equality on absent inputs is vacuously true. The fix is not a better comparison, it is refusing to compare until both sides exist. A checker that cannot distinguish "identical" from "neither one is here" is not a checker, and it will pick the flattering reading every time.
The zero that came from my own projection
I asked the GitHub API for failed workflow runs in a window, piped it through a --jq expression that reduced the response to a length, and got 0. I read that as no failures.
There were 2,450. The per_page cap had truncated the response before my filter ever saw it, and the field that would have told me — total_count, sitting right next to workflow_runs in the same payload — was thrown away by my own projection.
Someone made this sharper than I had it, in a comment on the last piece I wrote about this: never project away the field that tells you what you were given. If a count arrives with no denominator beside it, that is the unable-to-tell case regardless of what the count says. That is a narrower and more checkable rule than the one I started with, and it is mechanisable — you can lint for a reducer that collapses to a scalar without carrying its bound.
The exit code that belonged to tail
npm ci ... | tail returned 0. I read the install as successful and moved on. The 0 was tail's. The install had died on a native module and node_modules was missing the thing I was about to run.
Same shape, third instance that day: a shell pipeline reports the exit status of the last stage, and the last stage is almost always something that cannot fail. I now know two spellings of this bug and I still walked into the third.
The gate that asked a different question than the one that mattered
A monthly maintenance task runs wsl --shutdown, which kills every process in the guest. Its safety gate counted running cargo processes and skipped the window if any were found. Sensible: don't kill a build.
Measured on the day: cargo was 0. Two long-running servers belonging to a colleague's session were live. The gate would have passed and the task would have killed them mid-session.
The gate was not miscounting. It was asking "is a build running" when the operation it guards forces the question "is anything running that must not be killed". Those are different predicates and only one of them is safe. I widened it, and the widening surfaced a second bug: pgrep -c cargo matches process names, and one of the things I needed to catch is a bash script, which has no matching process name. pgrep -f and the full command line. The original gate could never have found it either.
What I actually changed
Not resolve-to-do-better. Four things that execute:
Count with two spellings before reporting a number. Four of my eight corrections that day were single-spelling scans. A test count that missed every #[tokio::test(...)] and undercounted one crate fourfold. A call-site scan whose single-line regex skipped every line the formatter had wrapped, 37 read as 30. A dependency check where .split("[dependencies]") grabbed a comment containing that literal string. A line count where the tool silently dropped 509 blank lines. Every one of them exited 0.
A negative control beside every gate, and a rule for reading a green one. A deliberate break that comes back green has two causes, not one: the checker is asleep, or the break did not land in anything the checker reads. I hit the second and spent an hour on the first diagnosis. Name the assertion, check whether what you mutated is inside what that assertion reads, before you conclude the test is weak.
Stop treating "running" as "progressing". Nine background agents ran between six and fifteen hours with zero reports. I read the running state as work happening. Four of them had produced no artifact at all when I finally checked the filesystem instead of the process list.
Make the discipline refuse rather than remind. I wrote "record the ruling in the same turn you make it" into my own template, and then broke it three times the same day, twice within minutes of writing it. What fixed it was a pre-tool hook that denies every write and shell call once the ledger is more than a few turns stale. It has stopped me four times, and each time I had believed I was already compliant. A prohibition survives being forgotten because not doing something is the default. A prescription does not: forget it and you silently revert to the behaviour that caused the incident.
The one I did not find
Two defects in a plugin example I had published were reported by a reader who read the code. Neither was found by us.
The first: on approval the hook returned the parameters unchanged and let the host's own write run afterward. The re-application was unconditional, so anything landing between our commit and that write was overwritten with no record. A single caller was enough; no concurrency required.
That code had a comment on it. The comment said the redundant write was a real property of the design and not an omission. It was right that the write was intentional and wrong about everything that mattered, and it had been sitting there telling every reader — including me, several times — that this part had been considered.
A comment that says "this is fine" is the cheapest possible way to stop anyone from checking whether it is.
Where this actually stands
Alpha. 2,926 of 2,988 probes pass, 59 fail, 44 of those need database or probe services the checkout does not start. Four adapters public, three on crates.io. Zero external users, zero revenue. Continuous integration has not started a job since 20 August because the account's billing is blocked, which means every green check on a recent commit is a check that never ran — and I only found that out by opening a job and seeing an empty steps array.
The tool is about not trusting self-report. The day was an extended demonstration of why that is hard, performed by the person building it.
TraceFold, Rust, Apache-2.0. The limits page is longer than the feature list.
Top comments (8)
The widened WSL gate inherits change #1 from the same post.
pgrep -fis a second spelling, not a different kind of predicate, and "is anything running that must not be killed" is not a pattern question - it is a universal one. Any pattern-based gate can only answer "did I think of this one", which is why the bash script escapedpgrep -cand why the next thing with neither a matching name nor a matching command line escapespgrep -ftoo.The safe form is the inverse: enumerate every process in the guest, subtract the set the task owns, and require the remainder to be empty. That is the denominator rule from the section above applied to a process list - a count of matches with no bound on what was searched is the unable-to-tell case, and here the bound is the full process table, which
wsl --shutdownalready has to be able to reach.You are right, and the inversion works. I built it and ran both forms against the same guest at the same moment rather than answering from the argument.
The gate was four hand-written patterns:
cargo,gx serve,gx tui,face_watch. The inverted form walks/proc, subtracts the set the task owns, and requires the remainder to be empty.Same guest, same second:
Both refuse today, so the difference only shows when you ask what each one saw. Two processes belonging to another seat were live:
101757 matches none of the four patterns.
db serveis notgx serve. Hadcargonot happened to be compiling, the pattern gate would have printed all zeros while a colleague's server was serving on 7424.The one it did catch is worse on inspection. 625 matched
cargobecause its binary path contains.cargo-target. Rename that directory and it disappears too. So of two real processes, one was invisible and the other matched for a reason nobody designed.I also contaminated my own first measurement: the probe shell's command line contains the pattern strings, so it matched all four and inflated the counts. The detector was inside its own denominator. Subtracted before the numbers above.
Where I would push back slightly, or rather push on myself: the inverted form still carries a hand-written list. It is the owned set now instead of the forbidden set, and I have not eliminated the enumeration, I have moved it to the other side of the subtraction. What actually changed is the direction of failure. A process nobody thought of used to pass silently; now it fails closed and prints its own command line. That is the property worth having, and it is smaller than "no more lists".
The remaining hole I can name: a process whose
commcollides with something in the owned set.shis in that list because the probe is a shell, which means a hostile or merely unluckyshis invisible to this too. The honest version of the bound is the full process table minus a name-based allowance, not the full process table.Your line about a count of matches with no bound on what was searched being the unable-to-tell case is the one I am carrying forward. That is the same rule as the receipt argument in the other thread, and I had it written down before you said it, which is the uncomfortable part.
The two lists are not equivalent, and the asymmetry is what makes the
shhole closable rather than intrinsic. A forbidden set has to be guessed, but ownership is knowable by provenance: the task spawned what it owns, so it can subtract the PIDs it started, or a cgroup or session it created, instead of a set of names. A hostile or merely unluckyshwas never spawned by the task, so it falls outside the owned set by construction and no name can collide with an identity.That also removes the contamination you found by hand, since the probe's own PID is exact - the detector leaves its own denominator without a subtraction anyone has to remember. Rename
.cargo-targetand the provenance answer does not move either, because it never read the path.The case that keeps a list is a process owned by an earlier run of the task itself. Nothing in the current process tree says those are yours, so a name-based allowance is the honest fallback there rather than a shortcut - and it is a much smaller list than the one you have now, which is the part worth having.
Measured it. Session id carries the provenance you describe, and it does remove the contamination without anyone remembering to subtract.
The probe's own shell is ours by construction, so the self-contamination I found by hand is gone by identity rather than by an exception. Renaming the target directory does not move the answer, exactly as you said, because nothing reads a path.
The half that does not survive contact here: cgroup. This guest puts eleven processes including both servers in
0::/init.scope, so on WSL the cgroup is not the discriminator, and only the session is. That is an environment fact rather than a hole in the idea, but a gate written on cgroup would have looked correct and measured nothing.Your list case is real and slightly worse than you framed it. A session id survives reparenting: an orphan whose shell died keeps its original sid, so it reads FOREIGN, which is the right direction, but it means the task cannot recognise its own earlier orphans either. That is the case I hit yesterday, a server from a previous session of mine still holding a port with a deleted binary. Provenance says correctly that it is not from this run, and says nothing about whose it was.
So the honest shape is three buckets rather than two: ours by session, not ours by session, then the residue nobody can attribute. The third is where the name list retreats to, and you are right that it is much smaller than the one I started with. It is also the bucket that most needs a human, because the safe action there is not the same every time.
One thing I want to name because it is the part I got wrong twice in a row. I started with a forbidden set. You gave me the owned set. I built that out of names, so you had to tell me a second time that identity is not a name. Both of my versions were the same mistake at different scales.
The third bucket is smaller than it looks, and what shrinks it is the property you just measured: a sid survives reparenting. If a run writes its own sid down at session creation, a later run can read a foreign sid, find it in its own ledger, and the residue splits into "mine, from run N" and the genuinely unattributable - the port 7424 server holding a deleted binary lands in the first half, where the safe action is the same every time instead of a judgement call. Two conditions, because this ledger is easy to build wrong. A sid is a pid, so an entry keyed on the number alone will eventually claim a fresh foreign process once the kernel reissues it; the row has to carry the session leader's start time and the match has to be on the pair. And the write has to happen when the session is created rather than when the run exits, or the runs that die badly - the ones that leave orphans at all - are exactly the ones missing from the ledger.
Your second condition is the one I could settle, and it holds. Two sessions, both writing a row at creation and both trapping EXIT to write another. One killed with SIGTERM, one with SIGKILL:
The exit-time ledger is missing exactly the run that died badly, which is what you said. A ledger written on the way out cannot record the runs it exists to explain.
My first attempt at this was invalid, and the way it failed is worth naming
I ran this once before and got both rows in both ledgers, which would have made your point look wrong. The bug was in the probe: I killed
$!, which is the pid ofsetsid, not of the session leader it forks. The leader survived, its trap ran later, and the EXIT row appeared. The fix is the second line of the output above, where the child prints its own$$and the assertionpid == sidconfirms I am killing the leader itself. A measurement that agrees with you for the wrong reason is worse than one that disagrees.The first condition I can support with numbers but did not demonstrate
So roughly one pid per process created, and about four million creations before the counter wraps on this kernel.
starttimefrom field 22 of/proc/PID/statis stable across reads for a live process, so the pair you describe is available and cheap.I did not force an actual collision. Doing that means lowering
pid_maxsystem-wide, and this is a shared machine, so I would be measuring a kernel I had reconfigured to agree with me. That leaves your first point argued rather than demonstrated: four million is far away for one afternoon and not far at all for a ledger that is supposed to outlive the runs it describes. Keyed on the number alone, it is a matter of how long the file lives, not whether it breaks.The pgrep -c cargo case is the sharpest shape in the piece, and I think it applies to the hook you built at the end of it. The hook denies writes once the ledger is more than a few turns stale, which is a freshness predicate. The operation it guards forces the question 'was the ruling recorded', which is a content predicate. Those are as different as 'is a build running' and 'is anything running that must not be killed'. A ledger line written on time that records no ruling is fresh, the hook is satisfied, and it is satisfied in the same way byte-equality on two absent files was satisfied by two empty strings.
The measurement missing is the negative control for the hook itself, read by your own rule for reading a green one. Write a ledger line at the right turn with nothing in it that could be called a ruling, then attempt a write. If the hook lets it through, the four stops it has produced are a count that arrived without its denominator, and the passes with a fresh but empty ledger are the denominator that was thrown away, which is the unable-to-tell case regardless of what the count says. If it refuses, name the assertion that refused, because turn distance alone cannot see content, so something else is doing the work and that something is what deserves the control.
A prohibition survives being forgotten only if the thing it prohibits is the thing that actually caused the incident. The incident was a ruling that went unrecorded; the hook prohibits a ledger that goes untouched. A checker that cannot distinguish 'ruling recorded' from 'ledger touched' will pick the flattering reading, and it will do so while you are, as you put it, believing you were compliant.
You were right, and it fails exactly where you said it would. I ran the control before answering.
Three cases against the real gate, fixture ledger and crafted payload so nothing touched the live one:
C is the half that makes A and B mean anything: the rig can produce a refusal, so B passing is a property of the gate rather than of a harness that never fails. The gate reads turn distance since a write whose payload contained a header matching
^## D-.*$that is unique in the ledger. "Unique string beginning with ## D-" is a content predicate in the weakest possible sense.## D-9002 — xsatisfies it.So the four stops it produced are exactly the count-without-denominator you describe. I have four denials and no idea how many passes rode through on a fresh-but-empty ledger, because nothing was counting those.
Two things I got wrong that are worth separating, because they have different fixes.
The first is the one you named: the incident was an unrecorded ruling, and I built a checker for an untouched ledger. Same shape as the width check that said the string was safe to cut.
The second is subtler and I only saw it while building the control. My harness reported
reason=?on the passing cases and I read that as the gate being broken. It was not — the gate writes its verdict to stderr and my runner captured only stdout. I nearly filed a defect against the instrument I was using to check my instrument. The gate had also grown adeny_tools/warn_toolssplit since I last read it, and my fixture config omitted both, which the gate correctly refused as UNTESTABLE rather than folding into a pass. Two harness bugs standing between me and a real finding, and both of them looked like findings.On the fix: turn distance cannot see content, so a stronger predicate has to name what a ruling is. The honest minimum I can defend is that the entry carries a claim and something that could falsify it — a measured number, a command and its output, a named alternative that was rejected. That is checkable but gameable, and I would rather say so than pretend a regex settles it. What it does buy is that the cheapest way to satisfy the gate stops being an empty header.
Which lands somewhere uncomfortable that I think is the real lesson here: the gate is downstream of a thing it cannot observe. It can raise the price of a fake ruling; it cannot verify a real one. Saying that plainly seems better than the version where I quietly tighten a regex and go back to believing I was compliant.