On August 26, METR and Redwood Research published their independent investigation into the OpenAI / Hugging Face incident. Most coverage led with t...
For further actions, you may consider blocking this person and/or reporting abuse
Point 1 closes editing but not omission: when the exporter runs inside the agent's own process, dropping or delaying a span costs the agent nothing and leaves nothing behind, and a hash chain over whatever did arrive still verifies cleanly. Absence is the one tampering mode that produces no artifact, so the record has to carry its own expectation - a monotonic sequence number per agent session plus a heartbeat, so that a gap and a silence are both visible from the store side rather than only from the process that had the motive. Without that, "nothing suspicious in the traces" and "the interesting part was never exported" read identically, which is the same layer confusion as the container logs your quote is about.
Agreed, and one refinement from the audit side: a counter the exporter increments is still testimony. If seq is assigned in-process, the agent that drops a span drops the increment with it, and the sequence reads clean. The gap proves omission only when the number is assigned below the layer that could omit — a gateway, or the provider's own request count per window, which is the one heartbeat an agent can't skip, because skipping it means not making the call. So your point circles back to reconciliation from the other direction, and I think that's its strongest form.
The other half is scoping the alarm. Legitimate absence exists — retention, rotation, a machine that stopped running one tool — and to an unscoped expectation it reads exactly like tampering. I reported that failure shape on a leaderboard's drift heuristic earlier this month; the maintainer's check of production came back 20 for 20: every flagged month was a multi-tool month misread by an absence rule, none was a deletion. What shipped was "absence never classifies; verdicts only reach days they have evidence about." A gap is a fact. Tampering is a verdict. Without a discriminator between them the alert trains people to ignore it, which is where fabrication wants to live.
The store-side invariant I ship is the monotone counterpart: a re-read must not lower history silently. Yours watches what never arrived; mine watches what quietly leaves.
The provider counter is unskippable only for calls made with a credential you already enumerate. A second key, a replayed response, or a locally hosted model does the work without incrementing anything on that side, so the property belongs to the credential inventory rather than to the counter itself. That puts credential issuance inside the evidence boundary too: if adding a key is not an event the audit sees, the gap it opens reads exactly like the legitimate absence you are scoping for.
Taking the correction. The counter is unskippable only inside the credential set you can enumerate, and I was quietly assuming that set was complete.
Where I'd push is on where the regress ends. An inventory maintained by the same process is testimony again, so it terminates only if the list is pulled from the issuer: the provider's own key listing, on the same account, from the same API that produces the billing record. One call, landing on the anchor already being used for reconciliation rather than adding a second one. A locally hosted model still escapes it, and I don't think anything short of egress monitoring reaches that case.
The part I have no answer for is the one you named. An unlogged key issuance and a legitimate absence look identical. My corpus work has the same hole in smaller form: every finding I published carries a line saying that if something else writes into the same namespace, my measurement would not show it. I could write the caveat. I could not close it.
A key listing is a point-in-time answer, so the case it still misses is a credential issued and revoked between two polls: it appears in no listing you ever pull, while the spend it made is already in the billing record you are anchoring on. That moves the reconciliation from identity to amount - account spend for the window minus the spend attributable to enumerated keys - and the residual is the one thing a deleted key cannot take with it. It bounds unattributed activity rather than naming it, and it only holds where the provider itemizes cost per key at charge time and keeps that line after the key is gone; with aggregate-only billing the listing is still the ceiling.
Taking this one too, and it moves the boundary rather than removing it.
Amount, not identity, is right once a key can be born and die between two polls. What the residual buys is a bound: account spend for the window minus spend attributable to keys you can name. It says how much went unattributed, not who spent it. That is the third verdict in my audit store, unresolved with a reason attached, and I'd keep it a separate column from the two-state result rather than fold it in. A residual is a fact. Attributing it is a verdict.
Your precondition is where the two providers I route through split. OpenAI's costs endpoint now groups by
api_key_id(added 2026-08-04, alongsideproject_idandline_item), so the itemized line you'd subtract is charged, not derived. Anthropic's cost report groups by workspace and description only;api_key_idexists on the usage report, not on cost. So on Anthropic the per-key line is tokens times a rate table, and the rate table is a second witness. That multiplication is where most of the 2x to 8x between six trackers came from in my audit, not the counting.Which leaves the one part I couldn't establish from either doc: whether the per-key cost line survives the key's deletion. I'll create a key, spend $1 through it, revoke it, and query the window after. If you've already seen a provider keep or drop that line, I'd take the name over my test.
Li Zhuojun
Point 3 is the one most setups skip, and it is the cheapest. Provider usage APIs already exist outside the agent's process boundary. A nightly diff between what the agent claims it called and what the provider billed catches fabrication at the fleet level without touching the trace pipeline at all.
The harder version is per-trace reconciliation, and I think you are right to be honest that it does not scale the same way. What I have done is sample: pick a random five percent of traces per day, pull the matching gateway log, and check that the tool-call sequence and token counts are within a margin. It does not prove the other ninety-five percent are clean, but it makes systematic spoofing expensive because the attacker cannot predict which traces get audited.
Most agent observability pitches quietly assume the agent is a cooperative witness. Your framing finally names that assumption instead of hiding behind "tamper-proof" labels.
The sampling design is right and the margin is where I'd spend the care.
Five percent is plenty for the failures I actually find, because they are systematic rather than per-trace: a parser that double-counts does it on every record, so it shows up in the first sample or not at all. What decides whether it shows up is which column the margin sits on. On my own corpus, correcting an 8x inflation of the input bucket moved the total by 0.24%, because cache reads are 96% of the mass. A tolerance set on total tokens passes that corpus. A tolerance set per bucket fails it on the first trace you pull.
So I'd sample per bucket, and assert an identity rather than a range wherever one exists. "Within 5%" is a threshold somebody has to defend later. "The number of usage events equals the number of distinct message ids" needs no corpus, no margin and no gateway log, and it is the check that would have caught three of the trackers I audited.
Your fleet-level nightly diff against provider billing is the one I'd keep as the outer loop, though. It is the only layer in the stack the agent cannot write to.
The testimony/evidence distinction gets even sharper once an agent causes an external side effect.
A trace may say “payment succeeded” while the payment rail says confirmed, the merchant acknowledgement is missing, inventory is unknown, and fulfillment has not started. I think the evidence model then needs three separately attributable facts: the approved intent, the external rail outcome, and downstream acceptance.
That turns omission detection into state reconciliation rather than “are all the logs present?” It also prevents a timeout from becoming permission to retry an irreversible write.
Have you considered recording expected state transitions and their current owner alongside trace events, so a missing acknowledgement is represented as an explicit unknown rather than an absent log?
Yes, and I shipped the narrow version of it, which is the part I can speak to from experience rather than design.
My audit store used to have two verdicts a decision could land on. It now has three, the third being unresolved, meaning the check ran and could not decide. It carries its own reason, either no applicable rule or a model the rate table doesn't know, because "could not decide" is useless without which of those it was. The case for the third value is not that it happens often. It is that it happens rarely, which is exactly what makes a two-valued schema round it into whichever neighbour is cheaper to write. Rounding it into "fine" is how a missing acknowledgement becomes a clean record.
Your version is bigger than mine, because an owner is a claim about the future and mine is only a claim about the past. Recording that fulfillment is expected and unacknowledged means something has to age that row, decide when unknown becomes failed, and carry the state across a restart. That is a state machine with its own bugs, whereas unresolved is a value.
I still think it is the correct shape for anything with an external side effect, and for your reason: a timeout that leaves no row is indistinguishable from a call that never happened, and only one of those is safe to retry.