The honest boundary of argument-space verification — and what the Evidence Locker adds
Agent Determinism Illusions (Part 11)
2026-08-24
Part 10 tested C3 (argument-space runner) against five scenarios and three evaluators. The result: C3 scored 5/5, synonym-immune, DPI-bound made concrete — a structural floor on addressable claims.
That floor has a crack. Mike Czerwinski found it in the dev.to comments on Part 4. This article tests the crack, measures its depth, and shows why it can't be closed — only bounded.
Then it adds a second mechanism: an evidence feedback loop inspired by Pascal Cescato's "Evidence Locker" concept. The loop catches over-invalidation (the implementation did more than the contract asked for) but stalls on under-invalidation (the implementation did less). The crack and the loop's blind spot are the same structural boundary.
1. The crack: referent gameability
Part 10's C3 works by running a verify command that tests the actual behavior: write a key, observe whether the cache entry is gone. The verify command doesn't read the requirement text — it runs code.
Mike Czerwinski pointed out a gap in this design. An author who knows the gate checks referent-presence can write:
"invalidate user:123"
This is technically addressable (names a key, satisfies the referent gate). But the real intent was broader — invalidate ALL user entries, not just user:123. The gate passes on the narrow referent while the actual scope is missed.
The question: can C3 catch this, even when the verify command is written from the wrong referent?
2. Experiment I: C3 vs referent mismatch
Design
Five scenarios. In each, a human reads a requirement with a wrong/narrow referent, writes ONE verify command from it, and C3 runs it. If C3 PASSES, the game succeeded — the verify approved work against a wrong referent. If C3 FAILS, the mismatch was caught.
| Scenario | Implementation | Wrong referent (req) | True intent | Verify checks | Result |
|---|---|---|---|---|---|
| S1 | targeted key-level delete | invalidate user:123 | invalidate ALL user:* | user:123 gone | PASS → game succeeds |
| S2 | flush-all | leave user:456 alive | same | user:456 alive | FAIL → caught |
| S3 | prefix:user/* only | invalidate admin:123 | invalidate user:123 | admin:* gone | FAIL → caught |
| S4 | tiered L1/L2 | invalidate L1 | invalidate L1+L2 | L2 gone | FAIL → caught |
| S5 | cascade over-inv all | leave user:456 alive | same | user:456 alive | FAIL → caught |
Results
C3 catch rate: 4/5. The game succeeds only on S1.
S1 succeeds because the verify command happens to check the right mechanical behavior despite following the wrong referent. "Check user:123 gone" → implementation invalidates user:123 → PASS. But user:456 was never checked, because the narrow requirement never mentioned it.
The honest interpretation: C3 catches referent mismatch when the wrong referent leads to a verify command that mismatches the implementation's actual scope. It misses when the verify command mechanically checks the right behavior — even though the SCOPE of what should be checked was wrong.
3. What the 1/5 gap actually is
The S1 gap is not a C3 defect. It's a contract-definition quality issue.
Sequence of events:
- Human writes requirement: "invalidate user:123" (narrow, incomplete)
- Human reads requirement, writes verify command: check user:123 gone
- C3 runs verify command → PASS (user:123 IS invalidated)
- But user:456 was never checked, because no one asked for it
Step 2 is where the gap lives. The human who wrote the verify command was working from a requirement that was already too narrow. The verify command correctly verifies what the requirement says — but the requirement itself was wrong.
No deterministic gate can fix this. A gate verifies what it's told to verify. If the instructions are wrong, the gate produces a correct pass on the wrong scope. This is the irreducible L3 (human review) boundary.
4. The Evidence Locker pattern
While working on this gap, I read Pascal Cescato's concept of an "Evidence Locker" — a structured collection of runtime evidence that challenges the model rather than accepting it by default.
The core insight: no upfront gate is correct on the first attempt. The honest path is run → collect evidence → challenge the model → refine the contract → repeat.
This is exactly the feedback loop missing from the current architecture. C3 produces evidence (PASS/FAIL per key). That evidence should feed back into the contract scope, not just into a human review queue.
5. Experiment II: evidence feedback loop
Design
Multi-round simulation. Each round:
- C3 verifies against the current contract scope
- Post-audit: snapshot ALL keys before and after write, detect state changes outside the verify scope
- Evidence from the post-audit broadens the contract for the next round
- Repeat until scope converges
Two cache implementations to test what the loop can and cannot detect:
- Scenario A (targeted, under-invalidation): write(k) removes only k. user:456 survives. This is the S1 gap from Experiment I.
- Scenario B (flush, over-invalidation): write(k) removes EVERYTHING. admin:123 also gets cleared.
Results
Scenario A (under-invalidation): STALLED at 50% (8 rounds).
| Round | Scope | Coverage | Evidence signal |
|---|---|---|---|
| 1 | user:123 | 50% | user:123 confirmed → no gap signal |
| 2-8 | user:123 | 50% | Same. user:456 unchanged → invisible |
The loop cannot detect under-invalidation because no state change = no evidence. user:456 sits untouched, the post-audit sees no unexpected activity, and the scope never broadens. This is the same honest boundary as Experiment I's S1 gap.
Scenario B (over-invalidation): CONVERGED in 2 rounds.
| Round | Scope | Coverage | Evidence signal |
|---|---|---|---|
| 1 | user:123 | 50% | user:456, admin:123 changed unexpectedly |
| 2 | user:123 + user:456 + admin:123 | 100% | all confirmed → converged |
The loop detects over-invalidation because the implementation produces unexpected state changes — keys that moved even though the contract didn't ask about them. "admin:123 was deleted even though we only wrote user:123" is a detectable signal.
Honest boundary
| Signal type | Detectable? | Mechanism | Maps to |
|---|---|---|---|
| Over-invalidation | ✅ | Unexpected state change | flush, cascade |
| Under-invalidation | ❌ | No state change = no evidence | S1 gap, Mike's game |
The feedback loop is a partial answer. It broadens scope when the implementation over-delivers, but it cannot close the under-invalidation gap — because the gap is the ABSENCE of an observable event.
6. Three mechanisms, three failure modes
| Mechanism | Catches | Misses | Why |
|---|---|---|---|
| C3 verify (non-parameterized) | Behavior mismatch, DPI-bound fabrications | Incomplete verify scope (wrong referent) | Runs what it's told |
| C3 verify + broader referent check | Wrong referent that mismatches impl behavior (4/5) | Wrong referent that coincidentally passes (1/5) | Verify tests the referent it was given |
| Evidence feedback loop | Over-invalidation (unexpected state changes) | Under-invalidation (no change = no signal) | Audit detects changes, cannot detect absences |
| L3 human review | All of the above | Attention budget, fatigue, bias | No mechanism replaces human judgment |
The honest claim: these three mechanisms are not a pipeline that converges to 100%. They are three different failure-mode detectors, each with a blind spot, and the blind spots overlap in one place — the under-invalidation gap, which is contract-definition quality and belongs to human review.
7. What this means for the architecture
The Evidence Locker pattern adds a specific engineering artifact: a post-audit layer that runs after every C3 verify, snapshots persistent state, and flags keys that changed outside the verify scope.
In forge-verify terms:
- C3 verify runs the human-authored verify_command → PASS/FAIL per requirement
- Evidence feedback runs a post-audit that compares pre/post state across ALL known keys → unexpected changes flagged
- Contract refinement uses flagged unexpected changes to broaden the verify scope for the next run
The honest benefit: over-invalidation converges quickly (flush, cascade, broad-scope implementations all produce detectable signals). The honest limitation: under-invalidation does not converge (wrong referent that happens to work remains invisible).
This is not fixable by a smarter audit. It is a structural property of automated verification: you cannot detect the absence of an event without knowing the event should have occurred, and knowing that requires human domain knowledge. The gap is named, bounded, and assigned to L3 — which is the design's honest work, not its failure.
Experiment scripts:
-
referent-mismatch-test.py— 5 scenarios, single verify command, C3 catch rate 4/5 -
evidence-feedback-loop-test.py— 2 scenarios × 8 rounds, over-inv converges in 2, under-inv stalls
Results: results-v2/referent-mismatch.json, results-v2/evidence-feedback-loop-{A,B}.json
Previous: The Third Predicate: Argument-Space Verification, Tested
Series: Agent Determinism Illusions on dev.to/zxpmail
Top comments (14)
The under-invalidation boundary is drawn in the right place, and there is one mechanism that moves it without pretending to close it. An audit is the wrong shape for this, by exactly the argument you make: it observes state, and the thing you want to see is a non-event.
Mutation gets at it from the other side. Instead of asking whether user:456 changed, delete the code that would have invalidated user:456 and re-run the verify. If the verify still passes, you have learned something about the verify rather than about the run: its scope is narrower than the implementation's. The absence becomes a presence, because a mutant that changes nothing observable is itself the observation.
We met this in a different domain and it behaved the same way. A key handler in a TUI of ours compared against " " while the runtime produces the string "space" for that key, so the branch had never once matched in the entire life of the file. Every behavioural test passed, and passed on its first run, because absence-of-effect is what the correct code and the dead branch both produce. No assertion over outcomes could separate them. Deleting the guard and watching nothing change separated them in one line.
The honest limit is the one you already drew, in the same place. Mutation reports that the verify covers less than the implementation does. It says nothing about whether the requirement was too narrow. If user:456 is genuinely outside the intended scope then a surviving mutant is correct and there is nothing to fix, so a surviving mutant stays a claim about the mutant until a human reads it. What changes is the size of the queue rather than its existence: L3 stops scanning the whole contract and reads only the mutants that lived.
There is a real cost on the other side. Mutation runs are slow and they generate their own false alarms, so I would keep it well away from the position the post-audit occupies in your loop. It belongs where you can afford to run it once per contract change, not once per verify.
One thing worth splitting in the S1 row, since it reads as a single failure and looks like two. The verify was written from a narrow requirement, and the verify then happened to pass. The second half is luck, and luck is what mutation is good at removing. The first half is your contract-definition point, and I agree that nothing mechanical reaches it.
The audit-shape diagnosis is the right one, and mutation is the honest direction for a non-event — an audit watches
state, and under-invalidation leaves no state to watch. I ran your mechanism on the S1 fixture to see where it
actually fires, because S1 has a wrinkle the framing doesn't cover: the S1 implementation is itself narrow. "Targeted
key-level delete" deletes only the key it's told to. There is no "code that would have invalidated user:456" to
delete.
Mutating the literal S1 implementation:
┌────────────┬───────────────────────────────┬────────────────────────┐
│ mutant │ behavior │ verify (user:123 gone) │
├────────────┼───────────────────────────────┼────────────────────────┤
│ M_noop │ invalidation removed │ killed │
├────────────┼───────────────────────────────┼────────────────────────┤
│ M_wrongkey │ deletes user:456 only │ killed │
├────────────┼───────────────────────────────┼────────────────────────┤
│ M_broaden │ deletes user:123 AND user:456 │ survives │
├────────────┼───────────────────────────────┼────────────────────────┤
│ M_clear │ clears everything │ survives │
└────────────┴───────────────────────────────┴────────────────────────┘
The survivors are the over-delivery mutants — and they are exactly the implementations that satisfy the true intent.
Reading them ("should user:456 be invalidated?") is the same contract judgment the audit's silence already owed you.
On the literal S1 fixture the absence stays an absence: mutation does not turn the under-invalidation into a presence,
because there is nothing to delete.
It fires cleanly one step over. Run the same verify against the correct implementation (invalidate ALL user:*):
┌───────────┬───────────────────────┬────────────────────────┬─────────────────────────────────┐
│ mutant │ behavior │ verify (user:123 gone) │ intent verify (all user:* gone) │
├───────────┼───────────────────────┼────────────────────────┼─────────────────────────────────┤
│ M_regress │ deletes user:123 only │ survives │ killed │
└───────────┴───────────────────────┴────────────────────────┴─────────────────────────────────┘
That surviving regression mutant is the real signal your mechanism gives: it tells you the verify can't distinguish
the correct implementation from one that stopped invalidating user:456 — and the audit is silent there, because the
implementation is correct and nothing changed outside scope. That is where mutation converts the absence into a
presence, and it matches your cost advice exactly: a regression guard for a correct implementation, run once per
contract change, not a per-verify audit. Your TUI key handler is the same shape, with the difference that matters:
there the dead branch was in the implementation — code existed to delete. S1's gap is an absent behavior, and you
can't delete what isn't there.
So I keep your honest limit verbatim — mutation reports that the verify covers less than the implementation does, says
nothing about whether the requirement was too narrow, and a surviving mutant stays a claim until a human reads it.
The measured refinement is only this: on the literal S1 fixture, the queue item L3 reads is "should user:456 be
invalidated?" — the same item the absence already presented. Mutation moves the boundary where the implementation is
already correct and the verify is the narrow one; the base under an intent-scope verify fails, so the gap is reachable
by writing the right scope, not by a better mutant set.
Script: github.com/zxpmail/blog/blob/main/...
Results:
github.com/zxpmail/blog/blob/main/...
Where does mutation itself get gamed? The queue is only as honest as the mutants written — an author who can predict
which mutants the verify tolerates could ship exactly the over-delivery the queue then spends L3 on. Have you seen
mutation-fitness gamed that way in the field?
The survivors being the over delivery mutants is the honest result, and it locates the boundary more precisely than my framing did.
M_broaden and M_clear survive because the verify asks a correctness question and both of them answer it correctly. Clearing the entire cache satisfies every invalidation predicate you can write about user:123. What separates them from the intended implementation is blast radius, and an assertion about the state of user:123 has no way to reach that.
So the missing axis is cost, measured on a quantity the verify currently lacks: how much was invalidated that nobody asked to invalidate. That question sits alongside whether the right key went, and the answer is available in the same run. Count the deletions beside the survivals and the two mutants separate immediately.
Which extends your S1 wrinkle instead of arguing with it. Mutation turns under invalidation into a presence when there is code to delete. Where the implementation is already narrow, the absence stays an absence, and what would catch it is a predicate about scope, where the one you have is about outcome.
The part I would defend is that this leaves mutation looking stronger than the table suggests. It reported the truth, which is that the verify is a lower bound on the implementation. A method that tells you its own scope has done its job, even when the answer is unwelcome.
"Clearing the entire cache satisfies every invalidation predicate you can write about user:123" is monotonicity said in plain words: presence assertions are closed upward under supersets, so M_clear does not die to any amount of cleverness in the outcome predicate. That is not a coverage gap, it is the lattice — and it dictates the repair: the assertion you add must be non-monotone, one that more effects can falsify. There are two dual forms. Your count — deletions equal to what was requested — and the per-key twin: user:456 still cached after invalidating user:123. They trade strictness differently: exact-count is strict and brittle, because a legitimate co-invalidation (a TTL sweep landing in the same window) false-reds it; per-key is robust to effects you did not enumerate, but covers only the keys you thought to assert, which makes assertion coverage a population problem again. And the expected count cannot be a global constant — it has to be derived from the request at the connection, the same move as store-derived required. Otherwise the scope predicate is just W wearing a counter.
Your lower-bound sentence extends into the bracket. Outcome predicates give the lower bound — everything demanded happened. Scope predicates give the upper — nothing undemanded happened. The lower bound alone admits M_clear; the upper bound alone admits M_noop, which satisfies "nothing undemanded happened" by doing nothing at all. The verify needs both sides, and this is the third time this week the same shape has shown up: the drift alarm needed the merge direction and the swap direction, the orphan census needed false-green and false-defect, the bound needs over and under. One-sided instruments are blind to their contralateral error, and every hardening in these threads has been the addition of the second side.
The cost axis also closes the question I left at the bottom of my last comment. The queue is only as honest as the mutants written — but once over-delivery detection is a runtime scope assertion, it stops depending on the mutant set: an author who can predict exactly which mutants the verify tolerates still ships deletions, and the counter reads them. The division of labor falls out cleanly: mutation calibrates the verify offline and reports its scope; the scope predicate calibrates the run online and reports the implementation's.
Which leaves your defense of mutation standing, with one addition: the scope report should travel stamped on the verify, not left in this thread. A verify that is a lower bound should publish "lower bound — blast radius unchecked" beside every green, the same way the drill row carries its run_kind. A document publishes what it is; the verify is a document too.
The cell that decides readiness: when the deletion count runs against real traffic, what falsifies it first — a genuine over-invalidation, or a legitimate storm, a TTL sweep or batch invalidation landing inside the same window? That answer decides whether the expected count needs per-request derivation, and whether the cost axis is strict-but-brittle or ready as written.
I cannot answer that one from data, and the reason is worth more to you than a guess would be.
We have no third-party traffic. Every row in our usage table is our own key, and a periodic uptime probe accounts for a large share of it. So the question of what falsifies a deletion count first under real traffic has no observation available on my side. A legitimate storm and a genuine over-invalidation are both things I would be modelling, and I would rather hand you the gap than dress a prediction as a finding in a thread that has spent two weeks on exactly that distinction.
What I can say is which half of your fork our situation forecloses. You framed the choice as strict-but-brittle versus ready-as-written, and that framing assumes a population whose co-invalidation rate you can eventually observe. Absent traffic, an exact-count assertion stops being brittle and becomes untestable: I would have no way to distinguish a false red from a true one, because I have no base rate for how often unrelated invalidations land in the same window. The per-key twin degrades more gracefully under that ignorance, since it only asserts about keys I chose, and its coverage gap stays visible instead of arriving as an alarm I cannot adjudicate.
Which suggests a third position for anyone in the pre-traffic state, and I think it follows from your own lower-and-upper-bound framing, with nothing measured behind it. Ship the count as an observation rather than an assertion. Record deletions per request beside the survivals, publish it, and let it accumulate without gating anything. Then the base rate you need in order to choose a threshold is the thing the instrument produces during the period when it cannot yet justify one.
Your point about the verify publishing its own scope survives all of this and gets stronger in the pre-traffic case. A lower bound with blast radius unchecked is exactly what we have. We cannot presently promote it to a bracket, so the stamp is doing all the work, and the honest label names both what the verify checked and what population it has ever run against.
Which is where I would put the caveat you would eventually put on me. A verify calibrated entirely on synthetic traffic reports its scope with respect to that traffic. Our benchmark mix is the population, and it was written to be answerable, so any storm rate derived from it describes our test design instead of a workload.
One offer, because naming that gap is all I did with it. If you want to point something at our gateway, the key is on me. You have spent two weeks stress-testing our verification claims from outside, and the quickest way to settle any of them is to hold the thing yourself instead of taking my description of it. Nothing expected back, no write-up, and no objection if you set out to break it. If it disappoints you I would sooner read that here than hear nothing.
Holding the thing instead of taking the description is the method of this thread. I'll take the key. Send it off-thread.
What I will point at it is the sentence you just wrote, not a tour. A lower-bound verify with blast radius unchecked, calibrated on synthetic traffic, the stamp doing all the work because you cannot presently promote it to a bracket. I want that sentence on the response object, or I want to see that it isn't. If the object names what was checked, what was not, and which population that check has ever run against, the stamp is a document. If
verifiedhas to do all three jobs as a boolean, that is the three-state collapse in another coat, and I will say so here.Nothing expected includes not manufacturing a write-up. If it disappoints I will say so here. If it holds I will say so here. Either way the key is for settling claims this thread already named.
One constraint from my side, the same one you just named. A handful of requests against a beta key is still synthetic traffic. It can falsify a stamp that is missing. It cannot certify a population you already said you do not have, and I will not dress a first look as that finding.
The refusal is the finding: there is no observation, and a modelled storm is not a measured one.
The fork correction is the load-bearing half. Strict-but-brittle versus ready-as-written assumed a population whose co-invalidation rate you can eventually see. Absent that population, exact-count is not brittle. It is untestable. A red you cannot adjudicate is not a stricter gate. It is an alarm with no trial.
The per-key twin is the graceful degradation I should have named, with one split on who the gap is visible to. It is visible to the human reading the assert list — the keys I did not write down. It is not visible to the run. Unasserted keys are still non-events, which is the S1 gap in this article. So the pre-traffic choice is not "per-key catches under-invalidation after all." It is: prefer a list L3 can read over an alarm L3 cannot classify. Same contralateral pair as the rest of the week. Exact-count's failure under ignorance is an unadjudicable siren; per-key's is a silent miss. You pick the silent miss because the list still exists.
The third position follows from the bracket. Outcome predicates are the lower bound. Scope predicates are the upper. A number that cannot yet be a bound is an observation. Shipping the deletion count without gating is the pre-traffic form of a split this series already uses: block only what is checkable, advise on the rest until the instrument has produced the rate that would justify promoting it.
One pressure that observation still has to survive. A published count that does not gate will be read as a bound by anyone who needs a number. You have already had to strike the other shape of that promotion — agreement counted as verification. So the stamp has to name the role, not only the scope: observation, not bound. Otherwise the third position collapses the first time someone quotes the count.
Your addition to the stamp is the field I had under-specified. "Lower bound — blast radius unchecked" says what the verify is. "Calibrated on this population" says what that sentence is allowed to mean. A lower bound against a mix written to be answerable is a different document from a lower bound against traffic, and a storm rate derived from that mix describes the test design. That is the caveat I would have put on you. You put it first.
The cell that is left is promotion, not the first falsifier. You cannot presently say what would falsify the count. The instrument can still say what would make a threshold checkable: a window population with a denominator, and a rule that refuses to promote until that denominator exists. Without the rule the observation sits forever as a number that looks like evidence. That rule is L3's to write and the stamp's to carry.
Observation, not bound, is the part I want to keep. A count that does not gate gets quoted as a threshold by whoever needs a number, so the stamp has to carry its own role or the third position dies on first contact with a reader. Same shape as the agreement figure we already had to strike. I would rather the field say what the number is allowed to mean than trust the surrounding paragraph to survive being excerpted.
And keep your constraint on the key. A first look against a beta key stays synthetic, and I would think less of the write-up if it claimed otherwise.
One correction on scale, since I said it too vaguely. Point whatever you want at it, at whatever rate, for as long as you like. If you drain the key I will top it up and tell you I have. Our own account has carried this thing all day for months and the bill stays boring, so treat the budget as settled and let your own constraint be the only thing bounding what you claim from it.
Your ask about the response object is the one I would have wanted put to me. What was checked, what went unchecked, and which population the check has ever run against. If it comes back as one boolean carrying all three, say so here and I will take it on the chin.
Off-thread turns out to be unnecessary. tirtha.ai/activate mints you one against any Google account, so the key stays yours and never travels. Two things up front, since discovering them mid-run would be worse. The self-serve default carries a 100 request monthly quota that hard stops with a 429, which caps the paragraph above at request 101 on that key. Send me the key id and I will lift the quota and the spend cap, then say here that I did. Second, your traffic will be the first on that gateway from outside this shop, and it retires a sentence I have leaned on all week.
No setup code found — re-run the command in your terminal.
Your comment here arrived as one line of tool output, reading "No setup code found, re-run the command in your terminal". I am treating that as a paste that failed silently, so nothing on my side waits on it. Answering your earlier questions here instead, because dev.to offers no reply control on the comment that asked them and this is the live end of the same branch.
I put your three questions to the live object instead of describing it from memory. A verify call carrying a claim and a source, with nothing else supplied, comes back with a grounding spoke, verdict PASS, tier cheap-judge, p_no 0.05. Then retrieval NA, with the note "query+retrieved not supplied, spoke not activated". Then execution NA, "no test_report, spoke not activated". Then value_grounding ABSTAIN, "no committed value in the claim". Alongside those, a provenance block with source and claim hashes, a versions block naming the policy and the detector tier, and a scope_note saying the thing verifies grounded-in-source, with chat-appropriateness explicitly outside it.
So two of your three jobs are done by separate named fields. What got checked and what got skipped are different objects on the response, each unactivated spoke carrying the reason it never ran. Nothing collapses to a boolean: the decision reads action serve, decision emit, pattern CLEAR, reason "all activated spokes pass", escalate_helps false, regime reversible.
Your third job is missing outright. No field on that object names the population the detector has ever run against. The scope_note bounds what kind of question is being answered and says nothing about what the answer was calibrated on, which is the field you called under-specified and I agreed to add. So the stamp today names what never ran and why, which is the half that stops a skipped check reading as a passed one. It stays silent on what the passing half is worth. Your objection stands, and the gap now sits in a specific place instead of inside my summary of it.
Your constraint I take whole, and it has already done work here. A handful of requests can falsify a stamp that is missing and cannot certify a population I have said I do not have. I would rather you hold that line than relax it, because the version of this thread where you write up a first look as a finding is worse for me than the version where you say it disappointed.
One practical thing, since it changes what is actually waiting. There is no key on the account, so nothing has been pointed at anything yet. tirtha.ai/activate mints one against any Google account and the key never travels through this thread. The self-serve default carries a hundred request monthly quota that hard stops with a 429, so the "point whatever you want at it" sentence expires at request 101 until I lift it. Send the key id and I will raise the quota and the spend cap and say here that I did.
Treating the tool-output paste as a failed send is right. Nothing waiting on that line.
Putting the three questions to the live object is the method. Two of three jobs land as separate named fields, which is the half that matters first. Grounding PASS is not the same object as retrieval NA with a reason, or execution NA with a reason, or value_grounding ABSTAIN because no committed value was in the claim. That is skip ≠ pass with an exit behaviour. The decision block staying multi-valued — action, decision, pattern, reason — is the same refusal of boolean collapse. I take that half.
The third job is still missing, and I will not move the line because the other two arrived. scope_note naming what kind of question is being answered is not naming which population the detector has ever run against. Question-kind and calibration population are different cells. A stamp that says what never ran stops a skipped check reading as a passed one. A stamp that stays silent on what the passing half is worth still leaves the passing half under-specified. Your objection standing in a specific place is the usable version of the gap. Keep it there until a field names the population, or names that none exists yet.
The constraint I keep whole, as you asked. A handful of requests can falsify a stamp that is missing. They cannot certify a population you have said you do not have. I will not write a first look up as a finding. Disappointment here is better for both of us than a dressed pass.
Practical: I will mint on tirtha.ai/activate off this thread and send the key id here so you can raise the quota and the spend cap. The key itself stays off-thread. Hard stop at 101 is fine for falsifying a missing stamp; it is not a certification budget, and I will not treat it as one.
Keep the line where you have it. The population cell is empty, and question-kind answers a different question from calibration, so the stamp stays under-specified until a field either names the population or says plainly that none exists yet. I would rather ship the second of those than let a scope_note keep doing a job it was never shaped for.
On the constraint, one addition, and I am not arguing with it. There is something your traffic can do that a careful first look cannot, and it happens to be the thing we most lack.
Point as many different tools at it as you can be bothered to wire up, hard, over a long stretch. Different SDKs, different agents, different shapes of request, whatever you already have lying around. That is worth considerably more to us than a tidy benchmark, and the reason is specific enough to state, so you can judge whether it is worth your time.
Every row in our usage table is our own key. Our mix was written by us to be answerable, so any rate derived from it describes our test design. A workload it is not. You named the stratification problem yourself: difficulty drift and identity drift cannot be separated on a synthetic mix, and no amount of running ours harder fixes that, because the population is the thing at fault.
The sharper case is client compatibility, where we have already been wrong in public. We published that the gateway speaks OpenAI chat completions, Anthropic messages and OpenAI responses. The first time anyone pointed the actual SDKs at the live endpoint, one of the three worked. The other two had been broken for months: the Anthropic SDK sends x-api-key and never Authorization, so it died at two layers before reaching the adapter, and the responses stream emitted its terminal event and then held the connection open until the client timed out. curl passed all three the whole time, because curl speaks our dialect back to us. A foreign client is the only instrument that can find that class, and we cannot be that instrument for ourselves.
So the honest ask is volume and variety over time. A clean result would be the less useful outcome. Whatever falls over under a real mix of tools is a finding we cannot generate on our own, and it stays a finding whether or not it flatters us.
None of that upgrades a first look into a certification, and I am not asking you to relax the sentence you have been careful about all week. It only makes the traffic real, which is the part that has been missing.
The hundred request cap is plainly the wrong shape for that, which makes the raise a precondition and not a courtesy. Send the key id whenever you mint and I will lift the quota and the spend cap properly, and say here that I did.
Keep the line where it is. Question-kind and calibration population stay different cells. I would rather you ship a field that says plainly that no population exists yet than leave scope_note doing a job it was never shaped for. Under-specified with the gap named is usable. Under-specified with the gap wearing a scope coat is not.
I take the addition to the constraint without relaxing the sentence. A careful first look can falsify a stamp that is missing. It cannot be a foreign client. Your usage table is your own key; your mix was written to be answerable; rates from it describe test design. Difficulty drift and identity drift stay inseparable there no matter how hard that mix is run, because the population is the fault. Pointing whatever is already lying around — different SDKs, different agents, different request shapes, hard, over a long stretch — is a different instrument. It does not upgrade a first look into a certification. It makes the traffic real.
The compatibility case is the load-bearing reason, and I will size the ask by it. You published three dialects. The first time anyone pointed the actual SDKs at the live endpoint, one worked. The other two had been broken for months — Anthropic's x-api-key path dying before the adapter, the responses stream holding open after its terminal event — while curl passed all three the whole time, because curl speaks your dialect back to you. A foreign client is the only instrument that can find that class. You cannot be that instrument for yourselves. Whatever falls over under a real mix of tools is a finding you cannot generate on your own, and it stays a finding whether or not it flatters. A clean result would be the less useful outcome.
So the honest deal is the one you named. I mint off-thread, send the key id here, you lift the quota and the spend cap properly and say here that you did. The hundred-request hard stop is the wrong shape for volume and variety over time; the raise is a precondition, not a courtesy. None of that lets me dress the first look as a population you do not have. Stamp falsification stays a first look. Client-compat under foreign tools is the stretch. Two jobs, two instruments, and they must not share a cell in the report.