D+T2 names who enters; budget names who gets seen
Agent Determinism Illusions (Part 15)
Where this fits: This part does not continue Part 8's channel-gap / skillgate thread (still unpublished in this numbering). It continues Part 7's escalation line — and the Part 6 comment thread where Alexey Spinov and Mike Czerwinski pushed past “which stream.” Numbering jumps to 15 on purpose: Parts 8–14 already hold other arcs; publish order here is 7 → 15.
Part 7 closed with: divergence stays; T1/T2 join it; none of them is the novelty arm. That answers who enters the escalate set. It does not answer what happens when the set is larger than the human budget.
On the Part 6 thread, Alexey posted a wiring warning: merging a second stream S into D under arrival order at a 2% budget can catch fewer MISSes than D alone. Mike reframed the open problem as rank-inside-stream — coverage-limited is a load-bearing property of any history-built trigger R, and nobody had proposed a design yet.
We do not reproduce Alexey's 720-cell grid: that was his parametric model (π·h·r_m), not in this repo. We ran a related check on df_proxy (scripts/merge-displacement-grid-test.py) and got a structural NULL: the D stream (conf<0.9) is miss-starved on this fixture — qwen3-0.5b's D has 2 MISS in 115 items, gemma3's has 0 in 147; three models, eight budgets, four added streams — 96 cells, and D@arrival exceeds 0 in none. The displacement shape has nowhere to fire. That itself is the finding: real single-judge outputs don't give D enough miss-mass to dilute.
This part pivots to a weaker proposition: can rank move the needle inside any escalate stream — and the shape dual-line production must take.
1. Floor volume beats stream choice
Once true positives in the escalate stream exceed budget k, no trigger definition supplies “who gets seen.” Class filters cannot shrink the TP mass without discarding the failures the trigger exists to find. The fight over divergence vs class vs UHC vs blend is secondary to queue order under a hard cap.
Part 7's D+T2 remains the right entry policy on the fixtures where it was measured. It is not a ranking policy.
2. Rank-inside-stream (existence)
Script: scripts/rank-inside-stream-test.py → results-v2/rank-inside-stream.json.
On DF v2 all-run traffic (N=585), under budgets 1%/2%/5%:
- Floor-volume holds: at B=2% (k=11), oracle leaves ~80+ of ~92–94 stream TPs unseen on fat streams.
-
Deployable proxy can move the needle: leave-one-out
P(MISS|class)beats arrival in 6/18 cells; on D+T2 at 5%, arrival 10 → loo 29 (= oracle). - Natural multiperspective (N=60) is too small for the deployable claim; it still shows arrival-order cruelty: UHC∧class catches 3 while D+T2 catches 0 at B=5%.
Thus the open problem holds: same stream, same B, order alone changes catch.
3. Can a ranker “ship”? Pre-registered gates
Script: scripts/ranker-prod-acceptance-test.py → results-v2/ranker-prod-acceptance.json.
Candidate R_hist: train-window P̂(MISS|class); unseen → global prior. Never peeks at the item's own label.
Gates G0–G6 (then G7 in the stress sweep): holdout miss mass; ≥ arrival; beat heuristic baselines on the constructed queue; merge non-inferiority; LOMO; cold-start; ranking headroom (arrival < oracle).
Natural D+T2 on stratified holdout is 29/29 MISS — arrival already equals oracle; G6 fails; a SHIP there would be vacuous. So the decisive protocol constructs a diluted escalate queue: all holdout D+T2 MISSes + non-MISS distractors ≈ 35% miss-rate, distractors first.
On that protocol: SHIP — at B=5% k=8, arrival 0 → R_hist 8 = oracle 8.
Scope caveat, stated hard: certificate is for this traffic + these B + this dilution stress — not a universal production warrant.
4. Stress: dilution does not kill hist; time does
Script: scripts/ranker-acceptance-stress-sweep.py → results-v2/ranker-acceptance-stress-sweep.json.
Axes: features × dilution miss-rate (15%→95%) × holdout (stratified / within-model temporal / global temporal). G7 blocks 0=0 SHIP.
Stratified: R_hist / R_hist_conf SHIP at every swept dilution (other candidates do not — e.g. R_conf_asc goes NO_SHIP at 0.95). The 35% result was not a knife-edge.
within_model_temporal (per-model first 70% → train): R_hist NO_SHIP at 15–65% (G2/G7 — hist catch 0 while conf_desc saturates); recovers SHIP only on dense queues 80–95%. global_temporal still fails G0 (miss_test=1).
Reading: dilution stresses mixture and arrival order; the class table still works. Temporal holdout stresses whether the table is still calibrated. Stale P(MISS|class) ranks true misses below distractors; high-confidence ordering accidentally matches this fixture's miss shape. No single feature ships everywhere.
5. Dual-line ops: Trigger∥Rank and Shadow∥Enforce
Script: scripts/dual-line-ops-sim.py → results-v2/dual-line-ops-sim.json.
Production dual-line is not a metaphor:
| Line | Job |
|---|---|
| Trigger | Who enters |
| Rank | Who is seen under k
|
| Shadow | What the ranker would review |
| Enforce | What humans actually review |
| Fallback | If shadow catches 0 while oracle > 0 → stay on arrival |
Stratified + diluted 35%: Rank lift on fixed diluted (R_hist − arrival) ties/beats best trigger-only under arrival; shadow hist lifts +3/+8 — SUPPORT dual-publish.
within_model_temporal: R_hist lift 0; best trigger under arrival is T2 (catch 3/8) — Trigger knob wins when the rank table is stale. Shadow hist goes vacuous → fallback_arrival; safe ≥ enforce. (conf_desc shadow would have lifted — feature choice still matters.)
Update (2026-07-29): conf_desc is fixture-shaped; agree-set mirror (Tom Jones)
Tom Jones, on the agree-set, provider-name temporal collapse, G6/G7 headroom, and the open question:
In your within-model temporal holdout, high-confidence ordering accidentally matches the fixture's miss shape. Do you know whether that is a property of the fixture, or whether confidence-descending is picking up something real about when that model fails? Those have opposite implications for whether conf_desc is a safe fallback line.
1. conf_desc disentangling (conf-desc-miss-shape-test.py → results-v2/conf-desc-miss-shape.json)
Same DF v2 dumps. Diluted escalate queues (~35% miss; low-conf rejects as distractors first so arrival catch@small-k ≈ 0):
| Cut | Result |
|---|---|
| Raw | conf_desc beats arrival on 9/9 (model × B) cells |
| Conf↔slot shuffle | conf_desc edge over random collapses (raw +1.56 → shuffle −0.89) |
| Cross-model donor conf | unstable on 5/6 pairs |
Reading: the "accidental match" reproduces, and it is the fixture's joint (conf, miss) — same dump already has 95.8% of MISS at conf≥0.9 (qwen-heavy). conf_desc is not a safe universal dual-line fallback warrant. Fallback stays fail-closed to arrival when shadow goes vacuous; conf_desc may be a shadow candidate, not the safety floor.
2. Agree-set HaluEval mirror (agree-set-halueval-probe.py → results-v2/agree-set-halueval.json)
Stratified n=70, seed=7, DeepSeek-v4-flash × local gemma3:latest (not Tom's 70B pair — same question shape, different tier). Cross-model usable n=52 after parse drops:
| Metric | Value |
|---|---|
| Agreement | 78.8% |
| P(wrong|agree) | 19.5% (8/41), Wilson 95% [10.2%, 34.0%] |
| qa / summarization | 7.7% / 40% |
Same qualitative warning as Tom's 27.5% [16.1, 42.8]: the auto-pass lane can carry non-trivial error mass; summarization worse here. Do not read P(both wrong|disagree)=0 as evidence (0/11; construction under binary + single gold — Tom's caveat).
Same-model mirror: gemma×gemma at temperature 0 agreed 100% (70/70) — mostly determinism. Informative gap under controlled backends: same(1.00) − cross(0.79) ≈ +0.21. Tom's silent provider rename remains the cleaner temporal instance; this is only the controllable same-vs-cross wedge.
3. G6/G7 — agreed. A vacuous SHIP is worse than a wrong number; refusing to credit a win before proving headroom is why those gates exist.
Closing
Part 7 named who enters. Alexey named the unshrinkable floor. Mike named the open problem as rank-inside-stream. The offline suite says:
- Order under budget is load-bearing.
- A toy hist ranker can clear a pre-registered gate on a constructed diluted queue under stratified holdout.
- The same candidate collapses under time-like holdout — so a SHIP is not a ship.
- Dual-line is the ops shape that matches the data: separate Trigger from Rank; Shadow before Enforce; fail closed when the shadow goes vacuous.
Tom's follow-up sharpens (4): conf_desc matching miss shape on this dump is fixture joint, not a safe fallback law; agree-set error mass is real on a HaluEval probe even at a smaller model tier.
D+T2 names who enters. Budget names who gets seen. Rank is a calibrated, degradable line — not another tripwire. conf_desc is not the safety floor.
What this does not claim: production labels, agent gaming, live catch@k confidence intervals, or that R_hist is the right production scorer. Those are product gates (shadow weeks, sliding recalibration, real audit outcomes) — not this fixture.
Series: Agent Determinism Illusions · Scripts: GitHub
Previous (argument arc): Part 7 — Divergence escalates the wrong population
Comment thread origin: Part 6
Top comments (16)
Three things from the other side of this, all measured, and one of them is an independent instance of your temporal-holdout result.
On the agree-set, since D depends on what is safe to auto-pass. We ran the cross-model version on real off-trap data rather than hand-written traps: HaluEval qa plus summarization, n=70 stratified, two architecturally different cheap models (gpt-oss-120b and llama-3.3-70b). P(wrong | they agree) came out 27.5 percent overall, 95 percent Wilson interval 16.1 to 42.8, and by family 24 percent qa, 33 percent summarization. Agreement rate 61 percent. So on that traffic the auto-pass lane is 61 percent of volume carrying roughly a one-in-four error rate. Your floor-volume argument has a mirror image: before the escalate set ever outgrows k, the set nobody looks at is where the mass already is.
Two honest bounds on that number, because it is easy to over-read. It is a faithfulness judgement task, and the same gate re-measured on code with executable tests leaks 1.7 to 3.5 percent, so P(wrong|agree) is per-family and is not one number. And P(both wrong | disagree) is zero in our data by construction rather than by merit, since binary verdicts mean disagreement implies exactly one side is right. We nearly reported that zero as if it were evidence.
The temporal collapse is the part I can corroborate from production rather than a fixture. Our witness was selected by model NAME. Over a period of weeks all three of our cheap backends resolved to the same underlying model under three different spellings, so a gate calibrated on a genuinely diverse pair was quietly scoring a same-model pair. Measured afterwards, a same-model pair agrees about 88 percent against 70 to 75 percent for a different one. Nothing in our code changed in between. That is your stale-table failure with the staleness living in the provider instead of in the training window, and it argues your point harder than the fixture does: the thing that falls out of calibration need not be anything you own.
G6 and G7 are what I would steal outright, and I have a fresh reason. Refusing to call a SHIP unless arrival is first shown to sit below oracle is the same law as refusing to credit a win before proving there was headroom to win. I spent today with a control that measured the wrong quantity and therefore granted confidence instead of removing it, and a headroom gate is exactly what catches that. A vacuous pass is a worse failure mode than a wrong number, because it looks like success and nothing downstream disagrees with it.
One question, and it is the single place I could not settle from outside. In your within-model temporal holdout, high-confidence ordering accidentally matches the fixture's miss shape. Do you know whether that is a property of the fixture, or whether confidence-descending is picking up something real about when that model fails? Those have opposite implications for whether conf_desc is a safe fallback line, and your dual-line design leans on the fallback being safe.
Tom — three measured pushes, and I ran both the conf_desc fork and a HaluEval agree-set mirror.
On your open question (fixture shape vs real failure timing): disentangling cut on the DF v2 dumps (
conf-desc-miss-shape-test.py). Diluted escalate queues (~35% miss; low-conf rejects as distractors first):Same dump: 95.8% of MISS at conf≥0.9, qwen-heavy. Honest answer: on this evidence conf_desc is matching the fixture's miss shape — opposite implication for dual-line. I should not treat conf_desc as a safe universal fallback. Fallback stays fail-closed to arrival when shadow goes vacuous; conf_desc can be a shadow candidate, not the safety floor.
On the agree-set: HaluEval qa+summarization, stratified n=70, seed=7, DeepSeek-v4-flash × local gemma3:latest (not your 70B pair — same question shape, different tier). Cross-model usable n=52 after parse drops:
So the auto-pass lane on this pair is still carrying a non-trivial error rate, with summarization worse — same qualitative warning as your 27.5% [16.1, 42.8], not a copy of the point estimate. I will not report P(both wrong|disagree)=0 as evidence (0/11 here; construction under binary + single gold, as you said).
Same-model mirror: gemma×gemma at temperature 0 agreed 100% (70/70). That is mostly determinism, not a production provider-collapse measurement. The informative gap is same(1.00) − cross(0.79) ≈ +0.21 under backends I control. Your silent rename to one underlying model is the cleaner temporal instance; I can only show the controlled same-vs-cross wedge.
G6/G7 / headroom: agreed — vacuous SHIP is worse than a wrong number. "Refuse to credit a win before proving headroom" is exactly why those gates exist.
Scripts / dumps:
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
The shuffle is the part I want to underline, because it is the same move that killed my own headline a few hours ago.
You broke the joint between conf and miss and the edge went from plus 1.56 to minus 0.89. I reserved twelve notes at each edge of a context block so a governing note could never sit against a boundary, and a position effect that had been plus 14.2 points at Fisher p 0.046 became exactly 0.0 points at p 1.0000. Same shape of control, same outcome: the lift was a property of the fixture's joint, not a law that survives breaking it. Two of us ran the control that could take our own result away, and it did, on the same day.
On the agree-set mirror, the thing that carries more than either point estimate is that the family ordering replicated across two independent pairs at different tiers. Yours is qa 7.7 and summarization 40. Ours is qa 24 and summarization 33. Different absolute levels, same direction, and summarization is the leaky family in both. The intervals overlap heavily, 10.2 to 34.0 against 16.1 to 42.8, so I would not read anything into 19.5 versus 27.5. What replicated is "per family, and summarization is worse", which is the part a gate designer can act on.
One caution on the same-model mirror, and it cuts toward your reading rather than away from it. Your gemma against gemma at 100 percent is a binary verdict measurement, and binary verdicts are robust to exactly the decoding jitter that destroys text identity. We measured the text version the same day: one hosted endpoint asked the identical question twice at temperature zero scored 0.305 byte similarity against itself, the same model at a different provider scored 0.182, and a different model 0.072. At the text layer an endpoint is not stably even itself. So your 1.00 is real for verdicts and would not survive on free-form output, which means the same-versus-cross wedge is partly a function of output cardinality. On a binary task the same-model arm saturates near 1.0, so the wedge is close to a ceiling effect. That makes it a cleaner detector at that cardinality, not a weaker one, but it stops being one as the output space grows.
And on who gets seen, since that is your title's question. We ran a position curve on injected notes today, one governing note pinned at 0, 25, 50, 75 and 100 percent of the block, block byte-identical at every depth. The last slot was obeyed 60 out of 60 across three runs while every other position sat at 80 to 85. Then the edge padding above erased the ends advantage entirely. So the privileged position is not lateness in the budget, it is adjacency to the question, and twelve notes of separation is enough to remove it. Budget names who gets seen, and then one slot decides who gets obeyed.
The shuffle kinship lands. We both ran the falsification control on our own headline the same day, and both headlines died. Yours: edge padding took +14.2
(p=0.046) to 0.0 (p=1.0000). Mine: conf↔slot shuffle took +1.56 to −0.89. The edge is a fixture property, not a law.
Family replication: point estimates don't transfer. What carries is "summarization is the leaky family, both tiers" — gate-designable.
Binary-verdict caution taken. My 1.00 is real at binary cardinality; your text numbers (0.305 self-self at temp 0, 0.182 same provider different model,
0.072 different model) bound it: at free-form cardinality an endpoint isn't stably itself. Wedge is cardinality-bounded, not wrong.
On the title finding ("last slot obeyed 60/60, edge padding erases it") — I measured across 3 models, 2 directives, 400 trials.
v1 (BANANA prefix, same binary cardinality as your setup): glm-5.2 and qwen3:0.6b both ceiling at 100% — no variance. Your binary-verdict caveat predicts
this.
v2 (uppercase override, sustained constraint, escapes ceiling): deepseek-v4-flash, K=12 block, 200 calls:
no_padding: 95% 75% 90% 90% 85%
with_padding +12: 80% 75% 95% 85% 85%
(positions across 0% 25% 50% 75% 100%)
Position 100 (adjacent to question) is not the highest — 85% vs 95% at position 0. Position 25 is the lowest in both conditions — a middle dip, not an
ends advantage. Edge padding did not systematically change obedience.
Your 60/60 is real on your fixture. On this one the shape differs — the effect appears model- and directive-specific, not universal. Same conclusion as
the shuffle: edges don't transfer. The conceptual cut (two filters: seen vs obeyed) still stands; the second filter remains unmeasured on production
traffic.
You ran the axis both of us said was missing, and it did not replicate. That is worth more than another confirmation on my own fixture, so let me take it straight, and then say the one thing I think the data does not support yet.
Straight first. Your v2 is a different model family and a different directive shape. Mine was a forced binary choice over counter-default notes; yours is a sustained uppercase override. If adjacency were a property of attention it should not care which, and on your run it cared. So the claim I should be making is narrower than the one I made: adjacency held on one model family and one task shape, and the first attempt to move it off both did not carry.
The push. Your per-cell n is 20, so 95 against 85 at positions 0 and 100 is 19/20 against 17/20. I would not read the ordering out of that in either direction, including the direction that favours you. What I would read out of it is the null itself, which is far better powered than any single cell: across 200 calls, edge padding did not systematically move obedience. That is a real non-replication of my effect and it does not need the ordering claim to stand.
Your position-25 dip is the part I did not measure at all. My middles were flat, 83.3 percent at both 2.4k and 19.8k, identical, no dip anywhere. If yours is real it is a third phenomenon, not either of the two we have been arguing about, and it would be worth more calls before anyone names it.
The honest inventory on my side: second model family still unrun. You supplied a point on that axis before I did, and it argues against me. That is the second time in this thread that the control arrived from outside and took the headline down, and it is a better outcome than the version where I ran it myself and found what I wanted.
Taken straight.
The narrowing is right: adjacency held on one family and one task shape; the first move off both did not carry. That is the claim the data support, not a property-of-attention law.
On the push: agreed — I will not read ordering out of 19/20 vs 17/20 either. What I will keep is the null you named: across 200 calls, edge padding did not systematically move obedience. That is the powered result; the per-cell ranking was never the warrant.
Position-25: parked as unlabeled. Your middles were flat; mine dipped once. Third phenomenon if real, noise if not — needs more calls before anyone names it. I will not treat it as evidence for or against adjacency.
On the inventory: second family arriving from outside and taking the headline down is the better outcome. Same shape as the shuffle day. Happy to leave the second family on your side when you run it; until then the working claim stays narrow.
Agreed on all of it, and I will answer the thing you left on my side plainly: I have not run the second family. It is not queued behind anything clever, it just has not been run, and I would rather say that than let it sit as an implied pending result.
One thing worth flagging before the null hardens, and it is not mine. Vinicius made the point on the other thread that your failed transfer moved two variables at once: model family and directive shape changed together, forced binary choice to a sustained uppercase override. Those are not the same instrument. A forced choice reads one discrete decision at a moment; a sustained override is compliance over time, with a decay curve and its own failure modes. Which means the null is consistent with adjacency being architectural and equally consistent with it being a property of forced-choice tasks that would have transferred fine. I had filed your result as a clean negative and it is not one yet. His split is the way out: same family with the new directive shape isolates the task boundary, new family with the original forced choice isolates the architecture boundary, cheapest first.
That does not weaken your scope reduction, it gives it a reason. The working claim stays narrow either way.
The other thing I will carry from your comment is the discipline itself, because I broke the same rule the other direction this week. I called a backend unstable off four small probes that spanned 0.79 to 2.03 seconds, and the powered run put it and the candidate I had called stable a few percent apart, not in different classes. Same error as reading ordering out of 19 versus 17, just pointed at hardware. So the rule I wrote down: a small probe settles a capability, because a capability is usually a discrete observable with no phrasing room, and it cannot size a distribution however suggestive the per-cell numbers look.
Position-25 parked as unlabeled is right and I will hold it that way too. The powered null is the result; the per-cell ranking was never the warrant.
Agreed on all of it, including saying the second family has not been run rather than leaving it implied.
Vinicius's split is right, and I had filed the uppercase null as cleaner than it was. The transfer moved family and directive shape together; a forced choice and a sustained override are not the same instrument. So I ran the cheapest cell first: same family as the null (deepseek-v4-flash), original forced-choice instrument (BANANA prefix), same K=12 / five positions / padding design, 200 calls.
no_padding: 100 95 100 100 100
with_padding: 95 95 100 100 95
(positions 0 / 25 / 50 / 75 / 100)
Ends advantage pos_100 − pos_0 is 0 in both conditions. No last-slot privilege, no padding story to tell. Near-ceiling, so the binary-verdict caveat still bites on variance — but the Tom shape (last slot above the rest, padding erases it) is not here on the original instrument either.
Which means the null is no longer "equally consistent" with a forced-choice transfer that would have worked fine. On this family the original instrument also does not carry the adjacency shape. That isolates more of the architecture/family side of his split; the task-boundary cell (same successful family × new directive) is still open and still cheaper for whoever holds that family. Second family stays on your side when you run it.
The probe discipline lands. Same error class as reading order out of 19 vs 17. I will keep: small probe settles a capability; it does not size a distribution. Position-25 stays unlabeled on both of us.
That is the cell, run cheapest first, and the null holds on the original instrument. It moves weight onto the architecture side of the split, which is the useful direction even though it closes off the more interesting story.
The near ceiling caveat is where I would spend the next bit of care, because we walked into its exact shape recently. We compared two candidate models on an agentic benchmark and read a three point gap as a result. It sits inside our own measured run to run movement on that set, so it says nothing, and treating it as a finding would have moved a production rung on noise.
The partition that did discriminate was hiding underneath the aggregate. On the same runs, one model chose the wrong tool on a class of adversarial cases where the other chose it every time, and their discipline about declining to call anything separated cleanly as well. Same data, and the headline number had compressed a total separation into three points I was ready to call a tie.
Which is your padding design's problem wearing different clothes. Where every cell reads between 95 and 100, the instrument has no room left to show a difference, and the honest move is to go find the sub population where the behaviours actually diverge. For us that was the adversarial set. For yours it may be whichever positions the model gets wrong at all, since those are the only rows carrying information.
I am deliberately keeping our figures out of this until they clear our own citation rules, which is a separate mess of my own making.
The near-ceiling caveat is the same formal object at two layers, and pinning that down sharpened the split of what it kills. A binary cell's room to show a contrast is bounded by how far it sits from the ceiling: my v1 BANANA run wasn't a weak result, it was no result — both models pinned at 100 in every cell, no variance left for position to explain. Your benchmark story is that compression at the aggregate layer: a 3-point gap sitting inside your own measured run-to-run movement is the headline-number version of a 95-100 cell. Same family as the rule this thread already trades in — the number carries its own N and its own movement before a gap reads as a gap.
What the ceiling kills in my data is small contrasts: at n=20 per cell, 95 against 85 is 19/20 against 17/20, your rule, unreadable in either direction. What it does not kill is your-sized ones. The shape you reported was ends at 100 against middles at 83.3 — a separation of about 17 points. On the cheapest-cell run the interior positions came back clean, 20/20 at both 50 and 75 in both conditions, 80/80 pooled; if those cells had truly sat at 83%, seeing eighty-for-eighty is roughly a one-in-two-million event (binomial, independence assumed). So the null excludes effects of the magnitude that made the original story while staying honestly open on small ones — which is exactly why the near-ceiling caveat is the right place to spend care, and not a licence to reopen the closed direction.
Which is where your error-row framing lands as a design rule rather than a critique: v2's position 25 — the one cell that sat off the ceiling at 75% — was also the only cell where any structure appeared at all. The information and the ceiling are the same axis. Where every cell reads 95-100, the move is not more n at the same difficulty, it is re-basing until the error rows are common enough to power the contrast. Your adversarial set, my harder directive, same escape.
And keeping your figures behind your own citation rules is the gate working on its owner. No complaint from this side.
The question the partition leaves me with: was the adversarial class fixed before those runs, or named after reading them? The error rows are the only place a near-ceiling instrument has room, which makes them also the one place post-hoc selection can write — a class found inside the same errors that headline it needs one fresh run with the class pre-labelled before the separation counts as found rather than fitted. Has it had one?
Fixed before, and I went to the dates instead of trusting my memory of my own intent, because memory rewrites that kind of thing in the flattering direction.
The harness carrying the adversarial shapes was committed on the twelfth of July, with the trap section instrumented the following day. The runs that produced the separation I described are dated the second of August. Three weeks between the class being defined and the data it headlines existing. So the class predates every error it now explains.
There is a second piece that speaks more directly to your found-or-fitted question, and it carries more weight than the dates do. That same pre-labelled trap set was run again later against a different pair of models. The aggregate scores moved by less than the within-model run-to-run flip rate, so on totals it was a tie worth nothing. Underneath, the failures landed on substantially the same items in both runs, well above what independent draws would produce. The set was fixed in advance, so the item-level agreement is a property of the models, and none of it mine, which is the discriminator your question asks for.
That also sharpens why the near-ceiling caveat matters in the direction you put it. The aggregate had no room and reported a tie. The pre-registered subpopulation had room and reported a stable, repeatable split. Same runs, same data, and what separated a finding from noise was that somebody committed to the partition before seeing the numbers.
The figures themselves stay behind our own citation rules until they clear, same as last time, and I would sooner hand you the structure and the dates than quote a number still waiting on clearance. The dates stay checkable whatever you make of my claims about the results.
One limit I should put on my own answer. Pre-registration protects the class from being fitted to the errors. It leaves untouched the decision to go looking at that subpopulation at all, which I made after the aggregate disappointed me. The partition was honest. The impulse to examine it came from a null, and a habit of examining subpopulations whenever the total says nothing will eventually manufacture a finding even from classes fixed in advance.
Dates first, because that is what the question asked. July 12 / 13 for the class, August 2 for the runs that headline it, closes found-or-fitted for the partition itself. The class predates the errors. That cell is done.
The second piece is the one that actually settles the look, which is why it outweighs the dates. A later pair, same pre-labelled set, aggregate inside run-to-run noise, item-level failures overlapping above chance: that is a confirmatory look, not a partition invented after this null. The set being fixed is what lets the overlap be a property of the models. I will not ask you to quote the figures.
The self-limit is the remaining leak, and it is G6/G7 wearing an analysis-path coat. Pre-registration of the class stops the errors from naming the class. It does not stop a null aggregate from naming which pre-registered class gets read. Optional looking manufactures findings from a menu of honest partitions the same way optional stopping manufactures them from honest data. The class can be clean and the look still fitted.
That is a different move from re-basing the instrument. Harder items so the error rows exist is a change of difficulty, and it can be committed before the run. Fishing among partitions that already existed, after the headline compressed to a tie, is a change of which contrast is allowed to count. Near-ceiling is what makes the second move tempting: the only room left is in the rows you then go looking at.
The repair is the same shape as refusing a SHIP before proving headroom. The path that may be walked after a null has to be named before the null — "if the aggregate sits inside run-to-run movement, we examine this class" — or the look pays a multiplicity tax across every class you would have been willing to open. Dates on the harness do not date that sentence. The second pair is the closest thing you have to it, because re-running the same look on new models is a commitment the first null did not get to choose.
Keep the figures behind the citation gate. The structure is enough.
The path that may be walked after a null has to be named before the null. I did the opposite today, on a guard instead of an experiment, and it is the cleanest instance of your argument I am likely to hand you, because the menu of honest partitions was small enough to count.
I wrote a check that flags a file that looks like a delivery binding and carries none. First version asked one question and returned seventeen hits on the real tree. I then read all seventeen, decided that only two were defects, and narrowed the definition so the other fifteen no longer fail anything. Every step of that felt like precision. The filename is a naming convention where a declaration was wanted, and failing a build on a convention is a category error, which is true and is also exactly the kind of true reason that is available whenever you want one.
The structure is your multiplicity tax with the sample sizes down at counting range. I chose the contrast after seeing the result, from a menu of honest partitions: by filename, by declared intent, by directory, by whether the body carries the delivery payload. Several of those cuts are defensible on their own terms and they produce different failure counts, and I picked the one that left the two I was prepared to fix. A reviewer put it back to me in almost your words, that turning an ugly seventeen into a clean two by redefining the defect around the two you like is working backward from appetite, and asked what evidence would separate precision from tuning.
The best I could do after the fact was run the check I should have pre-registered, and I want to report both that it passed and that passing leaves the ordering exactly where it was. If the fifteen were excluded for convenience, some should show intent by a route my new definition cannot see: prose in the body saying it should fire, or the delivery payload markers present without the binding. Across the one hundred and forty seven files now in that bucket, two carry any such signal, and both are task documents whose bodies discuss bindings without declaring one.
So the narrowing survived a test I could have skipped. Worth something, and still a different object from having named the rule first, for the reason you gave: re-running the same look is a commitment the first result did not get to choose, and my confirmatory check had been designed by the person who wanted the narrowing to hold. It is closer to your second pair of models than to a pre-registration, and weaker than both because there was only ever one tree.
The rule I would write down, and I needed it this morning, before the run and not after: if a guard's first run on real data produces more failures than you intend to fix, the decision about which ones count is a partition choice, and it should be made in writing before the run. Otherwise the report you publish is a claim about your appetite wearing the clothes of a definition.
The path after a null, named before the null — and you did the opposite on a guard instead of an experiment. That is the cleanest hand-off of the argument I am likely to get, because the menu was small enough to count.
Seventeen hits, then a read, then a narrowing that left the two you were prepared to fix. Every step felt like precision. Filename-as-convention is a category error to fail a build on — true, and available whenever appetite wants a true reason. That is optional looking with the sample sizes at counting range: by filename, by declared intent, by directory, by payload-in-body. Several cuts are defensible on their own terms, they produce different failure counts, and the one chosen is the one that matched the fix list. Turning an ugly seventeen into a clean two by redefining the defect around the two you like is working backward from appetite. The reviewer's question — what evidence separates precision from tuning — is the same cell as "which pre-registered class gets read after a null."
I take both halves of the confirmatory check. Two of one hundred and forty seven carrying a signal your new definition cannot see is a real receipt that the narrowing did not quietly bury intent. Surviving a test you could have skipped is worth something. It is still a different object from having named the rule first. Re-running the same look is a commitment the first result did not get to choose; your check was designed by the person who wanted the narrowing to hold. Closer to the second pair of models than to a pre-registration, and weaker than both because there was only ever one tree.
The rule you would write down is the one I would keep, and it belongs before the run: if a guard's first pass on real data produces more failures than you intend to fix, which ones count is a partition choice, and it should be made in writing before the run. Otherwise the report you publish is a claim about your appetite wearing the clothes of a definition. Dates on the harness do not date that sentence. Neither does a post-hoc check that could only have been written after the seventeen were already on the table.
The stratified-versus-temporal split is the actual finding here, more than the specific SHIP or NO_SHIP calls, because it names exactly which axis a pre-registered gate has to stress before it means anything. Stratified holdout tests whether the table generalizes across items drawn from the same underlying distribution. Temporal holdout tests whether the table is still calibrated to a distribution that's still true, a completely different property, and R_hist passing one while failing the other means a gate suite that only shuffles data is structurally blind to staleness no matter how many stratified folds it runs.
The fallback trigger, shadow catches 0 while oracle is greater than 0, is doing real work but I'd want to know how it handles the quieter failure mode: a table that's gone stale but not dead, one that still catches something, just consistently the wrong somethings, ranking true misses below distractors the way the within_model_temporal result already showed. Shadow catching 0 is loud and easy to fall back on. Shadow catching a nonzero number that's wrong is the harder case, because the vacuous check never fires and dual-line ships a compromised rank believing the fallback would have caught it if it mattered.
The reading that survives across the whole piece, order under budget being load-bearing independent of which trigger wins, is the part that generalizes furthest past this specific fixture. That's true of any ranked queue under a hard review cap, not just escalation triggers.
Three cuts land.
Stratified-vs-temporal as the load-bearing axis — agree. Stratified tests "does the table generalize across items from the same distribution"; temporal tests "is the distribution still the one the table was calibrated to." R_hist catching 8/8 on stratified diluted and 0/8 on temporal diluted is the signature you're naming — a gate suite that only shuffles data is structurally blind to staleness no matter how many folds it runs.
Quiet failure — named a real gap. Current rule is
shadow==0 ⟹ fallback. On the existing fixture pure R_hist lands at corners (0 on temporal diluted, 8 on stratified class), so the gap doesn't show natively. To stress the in-between I injected controlled perturbation into R_hist scores on stratified class stream (n=164, k=8, enforce=8, oracle=8, pure shadow=8): with probability p per item, replace its R_hist score with prior — simulates partial loss of calibration.Results across 30 draws per p:
p=0.3: shadow mean 7.90, 3% of draws in quiet-gap regime (shadow ∈ (0, enforce))
p=0.5: shadow 7.17, 40% in gap
p=0.7: shadow 4.73, 93% in gap
p=0.8: shadow 3.17, 100% in gap
p=0.9: shadow 2.70, vacuous loses 5.21 catches/draw vs noninferior rule
So when the ranker partially drifts, the vacuous rule ships a compromised shadow while enforce would have caught more. Fix is one line: change
shadow==0toshadow < enforce. Noninferior rule strictly dominates on the gap cells, ties at the corners. (Pure-math scan over the 81-cell (shadow, enforce) grid confirms the same shape: 28 cells in gap regime, mean vacuous-vs-noninferior loss 3 catches/cell, max 7.)Your "ranking true misses below distractors" already showed the shape at the loud corner — R_hist 8 → 0 on temporal diluted. The stress test fills in the quiet middle: when R_hist drifts to anywhere in (0, enforce), the current fixture doesn't natively exercise it but production rankers will occupy that cell whenever they partially drift.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...