Start with the thesis: Satisfied Is Not Established.
Technical source: R2.1 results, R3 results, R5 results, R3/R4/R5 reconciliation
Companion post: Context Is Part of an Agent's Authority
I built a benchmark family to test whether a governed metadata layer earns its cost when an agent selects enterprise context. I have now run it four times under four frozen contracts, redesigning the catalog and changing the acceptance ceiling along the way.
The claim that governance earns its cost against a cheap baseline has been rejected in every round that tested it. The round before those was rejected too, on a different rule.
| Round | Claim | Failed rule or controlling result | Verdict |
|---|---|---|---|
| R2.1 | Overall comparative claim | Governed holdout F1 0.24065, below the 0.245533 floor | REJECTED |
| R3 | Governed value vs. lexical | F1 CI [-0.371, 0.00005]; token ratio 2.11x vs. 1.10x ceiling |
REJECTED |
| R4 | Governed value vs. lexical | Quality passed; token ratio 9.86x vs. 3.0x ceiling | REJECTED |
| R5 | Governed value vs. lexical | Quality passed; token ratio 6.94x vs. 3.0x ceiling | REJECTED |
R3 through R5 are value-claim verdicts. R2.1 was the earlier overall-contract rejection that led me to separate the claims. Other claims passed: R3 and R5 accepted governed routing against full-context stuffing, while R4 returned those claims inconclusive.
Every round ran under a contract frozen before collection. The claim-scoped outcomes are documented publicly; R3 and R5 include public decision packs, while R4's later-derived pack remains held and is disclosed as such below.
R2.1 failed first. At its 3,000-object holdout the governed route scored 0.24065 against a prespecified floor of 0.245533, while the lexical prefilter scored 0.588.
Round 3: the simple baseline won the observed comparison
R3 compared three routes on the same local model: raw full-context stuffing, a cheap lexical prefilter, and a governed metadata route.
Background. This benchmark reproduces the structure of McKnight Consulting Group's study, "Stop the Token Bleed: Benchmarking the Benefits of Governed Metadata for Enterprise AI" (Jake Dolezal and William McKnight, August 2026; sponsored by Informatica, a Salesforce company). Their study held the model constant and found governed metadata access won on both cost (up to roughly 89x fewer tokens at scale) and accuracy (F1 1.000 against 0.29 to 0.66 ungoverned). Read their article for their full methodology and figures. This work does not reproduce their exact numbers. It lets you generate your own, on your own model.
Disclosure. The study reproduced here was sponsored by Informatica, a Salesforce company. I am employed by Salesforce. That is a reason to run this harness yourself rather than take my output on trust, which is the entire point of publishing it. Contradicting results are welcome.
Mean F1 across 20 seeds, at the prespecified 0% classifier-miss condition:
| Catalog size | Governed F1 | Full-context F1 | Lexical F1 |
|---|---|---|---|
| 300 | 0.780 | 0.261 | 0.660 |
| 1,500 | 0.632 | 0.253 | 0.737 |
| 3,000 (holdout) | 0.447 | 0.177 | 0.631 |
Read the two columns downward. The governed route degrades monotonically as the catalog grows: 0.780, then 0.632, then 0.447. The lexical route does not show the same pattern: 0.660, 0.737, then 0.631.
At the holdout, the keyword filter beat the governed route on the observed means and used less than half the prompt tokens. The paired F1 interval was [-0.371, 0.00005], which does not exclude zero in governance's favor. The token ratio was 2.11x against a frozen ceiling of 1.10x.
Governed routing crushed full-context stuffing. That claim passed. It lost to the cheapest thing in the room.
What I changed between rounds, said out loud
I ran R4 and R5 on redesigned catalogs, and I relaxed my own cost ceiling.
R3 used a lexically tractable synthetic catalog, where a keyword filter has real signal to match. R4 and R5 moved to semantic-access catalogs built on opaque physical names. In R5 the lexical route scored 0.000 F1 at every catalog size.
The redesign favored my method on quality: this name-only lexical baseline no longer had matching signal. But it also made the cost comparison harder, because the lexical route now produced an extremely small prompt. At the R5 holdout it averaged 110.0 prompt tokens against the governed route's 763.5, which is why the ratio rose to 6.94x even as the ceiling was relaxed.
Separately, between R3 and R4 I raised the maximum governed:lexical prompt-token ratio from 1.10x to 3.0x, which made the cost rule easier to pass.
Both changes were declared in new frozen contracts before their respective collections.
State that plainly, because a reader who diffs the contracts will find it anyway.
It still failed
In R5, governed selection used 6.94 times the lexical route's prompt tokens. The frozen ceiling was 3.0x. The preregistered claim that governance earns its cost against lexical filtering was rejected.
This happened on a task where the baseline scored zero. Governed context won the quality comparison but failed the prespecified prompt-token cost rule. The frozen contract required both.
R4 failed the same rule at 9.86x. R4 also returned INCONCLUSIVE on its full-context claims because one holdout request contained 128,256 input tokens plus a reserved 3,000-token completion budget, putting it 184 tokens beyond the verified 131,072-token window. It made no model call. It was retained as a preflight refusal rather than silently dropped.
The ceiling could have been relaxed again after seeing 6.94x. Moving a bar you already missed converts a result into a press release.
A benchmark that cannot reject its author is marketing with a methodology section.
What this changes for anyone buying or building a context layer
Ask three questions of any governed retrieval, semantic layer, or context-governance component.
What is the cheap baseline, and did you run it? Not merely full-context stuffing. At minimum, test the cheapest credible selective baseline: a keyword filter here, but potentially BM25, a cached lookup, or another simple retrieval route. If the only comparison is against full-context stuffing, the result may justify selective context, but it does not show that the sophisticated route earns its place over cheaper alternatives.
Was the acceptance rule written down before collection? A cost ceiling chosen after seeing the numbers is a description, not a test.
Where does the method lose? A vendor who cannot name the configuration where their layer is the wrong choice has not measured it hard enough.
My own answers: the baseline is a lexical prefilter, it beat the governed route on observed mean F1 and prompt-token use in R3, and the governed route has never cleared its cost bar in the three rounds that tested it. The route remains worth evaluating where opaque physical names make semantic selection necessary, or where an evidence trail has independent value. These runs establish the quality advantage in that narrow synthetic configuration; they do not yet establish end-to-end economic value. That is a narrower claim than the one I set out to prove.
Evidence boundary
These are synthetic, named-endpoint runtime characterizations on a single local model, qwen3-coder:30b. They are not production results, ROI claims, customer-data results, or a replication of any third-party study. R2.1 remains visibly rejected in the public record rather than buried.
R3 and R5 publish a full public packet: frozen contract hash, complete preflight, claim-scoped decision pack, and artifact digests. R4's contract is public, but its decision pack was derived after the fact from the archived report and is held rather than published. The raw reports stay private because they embed host identifiers, and their hashes are committed so you can tell if they ever change.
Four rounds in, the most useful thing this benchmark has produced is the boundary it refuses to cross.
Top comments (8)
The R5 boundary is the most interesting result here, and I'd separate two readings: the claim was rejected because it's weak, vs the instrument couldn't measure that regime. R5 looks like it sits on the second side — worth saying out loud, because it changes what you'd fix next.
When you redesign toward opaque physical names, the lexical baseline doesn't just lose matching signal — it produces an almost-empty prompt (110.0 tokens vs 763.5). At that point the token ratio stops measuring "governed metadata is more expensive than selection" and starts measuring "a route that returns nothing is cheaper than a route that returns something". Both statements are true; only the first was the claim under test. The frozen ceiling did its job — a pre-specified rule caught the difference and no one moved the bar — but a ratio ceiling is the wrong instrument for comparing routes whose outputs differ in size by an order of magnitude: it conflates selection quality with encoding verbosity.
The economic question is better asked budget-constrained. Fix the context budget (say X prompt tokens per request) and compare what each route extracts within it. Then "does governance earn its cost" becomes "given the same spend, does the governed route surface more of the relevant content" — which is the question a buyer actually faces. The quality advantage you measured in R5 might survive that comparison or might not, but the answer would no longer be dominated by how much the baseline chose to say.
The ratio comparison stays meaningful in the regime where both routes return comparable volumes — your R3, where lexical actually had signal to match. The failure mode is specifically the sparse-output baseline. Naming that distinction is exactly the kind of boundary this post is about, and it's a more precise takeaway than "governance is too expensive": in the name-only regime it's "the comparison needs a spend-equivalent frame".
Also: the R4 held-pack disclosure is the detail I'd keep front and center — a decision pack derived after the fact, disclosed as held rather than silently republished, is the fail-loud behavior most benchmark write-ups skip.
The distinction you are drawing is correct and the post does not draw it. It observes that the lexical route collapsed to 110.0 prompt tokens against the governed route's 763.5, and then reports the 6.94x rejection as though the ratio still measured what it measured in R3. Those are not the same measurement.
What makes it clearer than I put it: in R5 the lexical route scored 0.000 F1 at every catalog size. So the cost comparison is against a baseline that returns nothing usable. "Cheaper than a route that produces no correct answers" is not a finding about governance, and the pairing gives it away. R5 passed on quality and failed on cost against the same baseline in the same round.
So R5 identifies an instrument limit rather than supporting the intended cost-efficiency claim, and I should have said so instead of banking the rejection. The frozen ceiling behaved correctly, which is separate: a pre-specified rule fired and nobody moved it. It was also the wrong kind of rule for that regime, and those two things are both true.
Your budget-constrained frame is the fix. Fix the prompt budget and compare what each route surfaces inside it, which is the question a buyer actually asks. It also gives the sparse-output baseline nowhere to hide, because a route that returns almost nothing simply uses less of the budget and surfaces less.
The boundary you name is more precise than mine. Mine was "these runs do not establish end-to-end economic value." Yours is "the ratio comparison holds where both routes return comparable volumes, and breaks where the baseline is sparse." R3 is inside that boundary and R5 is outside it, which is a statement I can act on. That is the next round.
The 0.000 F1 detail does the work the 6.94x number could not: "passed on quality and failed on cost against the same baseline in the same round" is the cleanest statement of why R5 cannot be read as a cost finding. A route that returns no correct answers is not cheaper in any decision-relevant sense; it is absent.
Two things I would carry into the next round. First, the budget needs the same pre-registration you applied to the ceiling, or the original error repeats one level up. The honest sequence is: fix the budget, freeze it, hash the contract, then collect - because picking the budget after seeing R5's volumes is exactly the "moved the bar I just failed" move, applied to the comparison setup instead of the rule. Your R3/R4 story already established that discipline; it only has to extend to the new parameter.
Second, make the sparse case its own outcome class instead of a ratio data point. "Baseline returned 0.000 F1, surfaced nothing within budget" is a different result from "baseline surfaced the same content more slowly," and reporting them in one table invites exactly the conflation the frozen ceiling was built to catch. Three classes: governed wins within budget, baseline wins within budget, baseline did not participate.
And the line that survives the rewrite: R3 answered the cost question, R5 answered the instrument question, and the audit trail is what lets a reader tell them apart.
Both of those are corrections, and I am taking them.
The budget point is the sharper one, and it is uncomfortable because it is the same error I already wrote a post about. Picking the budget after seeing R5's volumes is moving the bar I just failed, applied one level up to the comparison setup instead of to the rule. The R3/R4 discipline does not transfer by being described; it transfers by being applied to the new parameter. Fix, freeze, hash, then collect.
On the three outcome classes, I think you are right and I can offer a piece of evidence for it from an unrelated direction.
I spent today on a security test suite with the same defect in a different costume. Thirteen verdicts were passing because a marker was absent — no credential leaked, no injection executed, no gold answer returned — against targets that had never exercised the control at all. One of them printed "Grader context is isolated from agent environment" against an endpoint that grades nothing. The fix was exactly your third class: not pass, not fail, "did not participate."
Same shape as "baseline returned 0.000 F1, surfaced nothing within budget." But the classification has to turn on participation, not on score. A route that ran and returned zero correct answers participated and failed. A route that could not produce an evaluable result under the precommitted budget did not participate in the scored comparison. Otherwise, the third class becomes a way to relabel weak results, which is worse than the ratio it replaced.
What that cost me is worth stating, because it is the argument against doing it casually. Every gate had to keep the ability to fail. A test that can only ever report "did not participate" is not a test, so each one ships with three seeded cases: one where it must pass, one where it must report that it did not participate, one where it must still fail. Your third class needs the same treatment, or it becomes a place results go to avoid being judged.
Your closing line captures the distinction better than I did in the post. R3 answered the cost question, R5 answered the instrument question, and the audit trail is what lets a reader tell them apart. I would like to use it in the next round, credited.
The seeded-cases guard is the piece I would have missed — a third class with no escape hatch becomes a place weak results go to hide, and "every gate keeps the ability to fail" is the correct antidote. Your security-suite story is a strong independent witness for the same shape one layer earlier in the pipeline: thirteen passing verdicts against targets that never exercised the control is the absent-marker defect, and "Grader context is isolated from agent environment" against an endpoint that grades nothing is the cleanest one-line example of a self-report that was never tested that I've seen.
The formulation I'd add to your "classification turns on participation, not on score": that makes the third class structural rather than editorial. "Did not participate" has to be decidable from the run's own record — was there an evaluable result under the precommitted budget — not from how the result feels afterward. Decide it before you look at the numbers, or the relabeling you're worried about becomes the default behavior rather than the abuse case.
Worth noting: a parallel thread on another article converged on the same fix from the type side — make absence of a result unrepresentable without a reason attached (a state with no default, so "nothing" cannot be emitted without saying which nothing it is). Participation-classification is that discipline applied to the comparison instead of the return type. Same instinct, two enforcement layers.
Credit is very welcome — no need to ask, and if it survives contact with your next round that is the best test it could get. Looking forward to the write-up.
"Decidable from the run's own record" is the formulation I was missing, and it is the one that makes the third class safe to have. Participation stops being a judgment about a result and becomes a predicate over the record: was there an evaluable result inside the frozen budget, yes or no. It is evaluated before any score is read. The predicate and its evaluation order are fixed in the hashed contract alongside the budget.
Your type-side convergence has a witness on my side too. The security suite's defect was not only that thirteen verdicts passed on an absent marker. It was that an unexercised control and a failed one were both recorded as the same value, and the difference lived in a sentence: a prose prefix in the details field said the control was never exercised. A serialised report carried the sentence but not its meaning, so any consumer had to re-implement a substring match to tell the two apart. The fix in that release was to make the state a field with its own predicate, not a phrase. Same instinct as your "nothing cannot be emitted without saying which nothing it is." The sentence could be read past. A required field can be validated.
The second release went one step further and is the part I would carry into the comparison. One verdict had literally written down that it could not test the property, and then passed:
That is the absent-surface case in its purest form, and it is now the third class. But the same change seeded a case where the surface is present and the property fails, and that case must still fail. The two ship together or neither is trustworthy.
So the next round pre-registers three things before collection, hash posted first: the prompt budget, the participation predicate over the run record, and the seeded runs. One forces the baseline to produce no evaluable result within the budget; the participation predicate must return false and the record must say "did not participate." Another produces evaluable but wrong answers and must be classified "failed." If the three-class scheme cannot distinguish those cases on demand, the third class is not a class; it is an exit.
Thank you, and the closing line goes in with credit to you.
Pre-registering the test rule before collection is harder than it sounds - we ran into this on an entity resolution pipeline where the threshold that 'passed' had been picked after seeing how the numbers landed, and we didn't realize it until the third run. The lexical filter beating the governed route on cost is actually the useful result here. It's what tells you when the complex approach is worth it, and that's something a number that moved its own ceiling can't give you.
Third run is about right. Mine was between R3 and R4.
I raised the maximum governed:lexical prompt-token ratio from 1.10x to 3.0x after R3 missed it at 2.11x. I had reasons, and the redesigned catalog was a real change, but the honest description is that I moved a bar I had just failed. The only thing that makes the change auditable is that the raise is disclosed in the post and each contract hash predates its corresponding collection. R5 then missed the relaxed ceiling too, at 6.94x, which is the outcome that makes the earlier move legible rather than convenient.
Your entity-resolution case is the sharper version, because a threshold picked after seeing the distribution looks identical to one picked before. Nothing in the artifact distinguishes them. Pre-registration is not the hard part. Not quietly re-deciding when the number lands wrong is the hard part, and it does not feel like cheating at the time.
On the lexical filter winning: agreed, and it is the result I would keep if I had to keep one. A cheap baseline that beats the expensive route tells you where not to spend, which is a decision you can act on. "Our method won" against a ceiling that moved tells you nothing you did not already believe.