Your Codex log file looked fine. Your SSD counters did not.
The file was small.
That was the trick.
If you opened ~/.codex/, you could...
For further actions, you may consider blocking this person and/or reporting abuse
That's quite a discovery! It's unsettling how something as ordinary as logging can quietly cause so much damage. A great reminder that even small things deserve attention. Excellent investigation! 👏
thank you! that is what made it so unsettling for me too. nothing looked broken from the outside, which is exactly why checking actual write activity matters.
Indeed, therein lies the crux of the matter.
The detail that makes this land is not the 37 TB — it is "the file stayed small, and that was the trick."
That is the whole class of bug. The instrument was working perfectly and answering a question nobody needed.
dureports allocated blocks,ls -lreports logical size, and neither of them is writes. The counter that actually wears the drive is on the other side of the device node — /sys/block/nvme0n1/stat field 3 (sectors written) or SMART Data Units Written — and nothing in a normal developer workflow surfaces it, which is exactly why it went unnoticed for months.The WAL point is what makes it structurally nasty rather than just noisy. A bounded SQLite database with a churning -wal companion is designed to keep the main file flat while writing constantly. So the shape of the file is not evidence about the shape of the I/O. Checking file size there is not a weak check, it is a check for a different question.
I have been burned by the same shape from the backup side. A pg_dump app | gzip > app.sql.gz reports gzip's exit status, not pg_dump's, so killing the dump a fifth of the way through gives you exit 0, a byte-valid gzip, and a pg_restore -l that proudly lists all seven tables — because the TOC is written at the start and the data phase is what got cut. Three checks green, none of them touching the thing that failed. I ended up writing up that whole family of false-positive checks; it is in my profile if this is your kind of rabbit hole.
The useful generalisation from your post: before trusting any metric, ask what its failure would look like. If the answer is "it would look exactly the same", it is not a check.
this is such a good way to put it. a check is only useful if the failure changes what it reports.
the pg_dump example is the same trap in a different costume. valid gzip, exit 0, readable TOC, and still no complete backup. everything around the actual failure looks healthy.
that is the question I want to carry into more tooling: if the thing we care about breaks, what would this metric actually do?
Your question is the right one to carry, and I would add a wrinkle: it has its own failure mode, which is answering it for a metric you just wrote and then assuming the answer holds for one you inherited. Same question, different answer, because the answer depends on which failure you imagine while asking it.
That is why I ended up asking it per failure mode rather than per metric. A metric with one failure mode gets one answer; one with three gets three, and they are not variations on a theme. The case I hit this week: /proc//io looks exactly like a per-process write counter, and I measured three separate ways it reports a confident number that is not the write cost. wchar counts bytes handed to write(), so a 64 MB write to tmpfs read 67,108,864 while write_bytes stayed at 0 -- nothing reached flash. cancelled_write_bytes refunds it: 256 MB written and then unlinked showed the full 268,435,456 against the device while netting zero. And the counters die with the process, so a 32 MB write inside a reaped child showed as nothing at the parent.
None of those throw an error, and that is the property I would now use as the test itself. Absence of a number looks like a bug. A wrong number looks like the world. So I would add to your question not only "what would this do if the thing broke" but "what would it still report if the thing never happened at all". If the answer is a plausible figure, the metric is measuring the instrument.
The reachability version of your question -- if the target were dead, could this check even tell me? -- I did end up packaging, because I kept needing it on machines that were not mine. It is small, local, and in my profile; no link, it either holds up or it does not.
Where I would push back on myself: I have not thought enough about the symmetric failure, the metric that moves when nothing broke. A false alarm spends attention, and attention is the budget every other check draws on. Has that one cost you more than the silent version?
yeah, false alarms have probably cost me more day to day.
not because they are worse than a silent failure in impact, but because they train people to treat alerts as background noise. after enough plausible but harmless numbers, the next real one arrives with the same tone and gets less attention.
the silent version is still scarier because you never get the chance to respond. but false alarms quietly spend the attention budget that would have caught it.
your “what would it still report if the thing never happened at all” question is the missing half. I’m going to start using both, especially for inherited metrics.
yeah. we had one this afternoon, same shape: bounce notifications were classified
as "unknown", and unknown means no push. so the one mail that tells you your
sending reputation is burning was the quietest thing in the inbox. weeks like that.
that is brutal. the system had a category for “we do not know what this is” and used it to make the message disappear.
those are the failures that make you stop trusting a green dashboard. hope you caught it early enough to limit the damage.
This is why I have trust issues
honestly, fair . In this AI world even i can't trust anything.....
The useful guardrail here is to monitor write amplification, not only file growth. A log can stay compact while rotation, SQLite/WAL churn, or repeated rewrites burn through flash endurance. I’d pair the SMART counter with a per-process write-rate budget and alert on a sustained slope change, then keep enough retention to correlate it with the agent or tool run that caused it.
Yep. One gotcha on the per-process part: the counters die with the process. A
32MB write in a reaped child showed as zero at the parent. We ended up on cgroup
v2 io.stat instead.
That is exactly the boundary that makes per-process counters misleading. cgroup io.stat gives the workload a stable identity across short-lived children; I would record the cgroup path and sampling interval with each alert so a later incident review can distinguish one bursty run from a sustained write pattern.
agreed on the interval — io.stat is cumulative, so a burst and a slow leak read
identical until you divide by elapsed time. what we got wrong is duller: the
cgroup path is in our logs but not in the alert, so the history can't be read
back afterwards.
yes, exactly. file growth tells you about retained data. the SMART counter and sustained per-process write rate tell you about wear.
a write-rate budget with a slope alert would make this visible much earlier, especially if it keeps enough context to connect the jump to a specific agent run or tool update.
The constant background indexing and snapshotting catches so many developers off guard. Even with modern high-TBW drives, seeing gigabytes of redundant temp writes every few hours is wild. Setting up dedicated tmpfs in memory or aggressive cleanup rules has become almost essential when running heavy assistant loops.
yeah, tmpfs can be a really good fit for disposable logs, indexes, and CI-style workspaces.
I would be careful with cleanup rules though. deleting files saves space, but rebuilding the same artifacts later can create just as much write activity again. the best win is keeping truly temporary churn out of persistent storage in the first place.
Very true, the other thing people dont really account for, is build artifacts. Not only do they bloat, but remember, when you have 100gb of build artifacts that you delete, that's 100gb of writes... Do that every 2-3 days, it adds up quite fast...
yeah, build artifacts are a big part of the same problem. one small correction though: deleting 100 GB usually does not write another 100 GB to the SSD. it mostly changes filesystem metadata and may issue TRIM commands.
the expensive part is rebuilding those 100 GB again, especially when the build creates lots of small temporary files, caches, and intermediate outputs. doing that every few days can add up very quickly.
That's what i mean, yes. Example, I am building a game as a hobby project, already at around 11m lines of code in rust. Dwarven Stronghold (like Dwarf Fortress, but expanded). Everytime it builds, it needs around 50gb of memory, which overflows my ram, so page-file eats at my ssd. Then the build artifact, which again, eats at the ssd, so because it makes me run out of space quite fast, I told the agent to run regular housekeeping, so that 100gb every day or so that it writes to disk, gets deleted and re-written with the next build.
Yeah, this is why if I WERE to use artificial intelligence to code, I would try only to use online versions.
that makes sense. using an online version moves most of the heavy work off your machine, though I would still keep an eye on browser storage and any local app or extension it installs.
the bigger lesson is just to check what a tool is doing locally instead of assuming a small folder means a small footprint.
Great article! One interesting point is that limiting database size through retention policies doesn't necessarily limit disk I/O or SSD wear.
I also noticed that, despite the fixes introduced in June, issue #43378 still reports excessive TRACE/DEBUG logging as recently as November 4.
From an architectural perspective, this reinforces an important lesson: observability needs resource budgets, not just retention policies. Reducing stored data isn't the same as reducing the cost of writing it.