Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · The Gate Is the Product · The Doctor Who Diagnosed Every Patient · 4 Models, 0 Promotable Edits
In v0.1.0, an edit fixed 4 tasks and broke 1. Net +3 on 26 tasks. p=0.23. Gate rejected. The ceiling was clear: if you do not move enough tasks, the gate should say no.
In v0.2.0, we expanded the A/B corpus to 40 tasks. We fixed the pipeline bugs. We added rejection context. We tested stronger models. The math got cleaner, not kinder.
The ceiling shifted. It did not disappear.
The v0.1.0 Result: 26 Tasks, 5 Movable, p=0.23
The edit was real. It fixed 4 tasks and broke 1:
| Task | Prompt A | Prompt B | Expected | Change |
|---|---|---|---|---|
| classify-015 | technical | urgent | urgent | FIXED |
| classify-023 | security | urgent, security | urgent, security | FIXED |
| classify-024 | feature | feature, billing | feature, billing | FIXED |
| classify-029 | feature | other | other | FIXED |
| classify-014 | technical | feature | technical | BROKEN |
Net: +3. Mean delta = 0.115. p=0.23. The permutation test computes this by shuffling task labels 1,000 times and counting how often random chance produces a delta ≥ 0.115. 23% of the time — above the 5% threshold.
The sign-test floor with 5 discordant pairs out of 26 is ~0.031 one-sided. Even a flawless edit that fixed all 5 would barely clear p<0.05 two-sided.
The v0.2.0 Result: 40 Tasks, Still Nothing
We expanded to 40 tasks. We should have more power. Here's what Mistral 24B, our strongest analyzer, produced:
| Iter | p-value | Mean delta | Accuracy |
|---|---|---|---|
| 1 | 0.55 | +0.025 | 64% |
| 2 | 0.52 | +0.025 | 64% |
| 3 | 1.0 | 0.0 | 64% |
| 4 | 0.52 | +0.025 | 64% |
| 5 | 0.77 | -0.025 | 64% |
Mistral produced positive deltas in 3 of 5 iterations. That is real signal. But the delta is +0.025 — 2.5% improvement on 40 tasks. At p~0.5, there is roughly a coin-flip chance this is noise.
The ceiling did not disappear. It moved: with 40 tasks, the sign-test floor for a flawless edit that moves 5 tasks is ~0.016 one-sided — clearable. But Mistral did not move 5 tasks with perfect accuracy. It moved roughly 1-2 tasks, directionally, inconsistently. A mean delta of +0.025 on 40 tasks is about one additional correct task.
Why More Tasks Didn't Solve the Problem
The naive assumption was: more tasks = more statistical power = easier for edits to clear the gate.
That's true, but it assumes the analyzer finds edits proportional to the task set. It didn't. The analyzer's edits still moved 1-3 tasks out of 40 — the same absolute count as 26 tasks. The effect size shrank from 11.5% to 2.5%. More tasks actually made the problem harder because the analyzer diluted its signal across a larger set without finding more movable tasks.
The statistical power argument works only if you hold effect size constant. We did not. The analyzer's effect size collapsed when we added more tasks. The bottleneck is not task count. It is the analyzer's inability to find edits that move enough tasks.
The Over-Correction Pattern: Still Alive in v0.2.0
The v0.1.0 edit had a side effect: it added "urgent" to tasks where the trigger keywords appeared but urgent didn't belong. The ExactMatch scorer punished it.
v0.2.0 introduced label-set-aware scorers. This helped — "urgent, security" no longer fails when the expected is "security, urgent." But the fundamental tension remains: any rule that adds labels will struggle under strict scoring.
Mistral's edits showed the same pattern. It kept trying to tighten urgency rules, which moved some tasks in the right direction but over-corrected on others. The analyzer never found an edit that was purely additive — no side effects, no regressions. The search space is too large and the analyzer has no exploration mechanism.
What Changed From v0.1.0 to v0.2.0
| Factor | v0.1.0 (26 tasks) | v0.2.0 (40 tasks) |
|---|---|---|
| Corpus size | 26 | 40 |
| Movable tasks per edit | 5 | 1-2 |
| Mean delta | 0.115 | 0.025 |
| p-value | 0.23 | 0.5+ |
| Gate result | reject (confidence) | reject (effect size, confidence) |
| Rejection context | none | wired into staged analyzer |
| Analyzer diversity | repeated same edit | Mistral explored variants |
| Promotable edit found? | No | No |
What I'd Tell Anyone Building a Self-Editing Loop
The statistical ceiling is real at any corpus size. More tasks help only if the analyzer finds edits proportional to the task set. It won't. The fix is better search, not more data.
A positive control oracle edit would have saved us months. If you hand the gate an edit that fixes every failure, and it still rejects, the problem is the task set. We never ran this test. We should have.
The permutation test is honest, not conservative. p=0.55 doesn't mean "maybe." It means "55% chance this is noise." Mistral's +0.025 mean delta at p=0.55 is the gate being transparent about weak evidence.
v0.2.0 proved the pipeline is correct. The math is the math. No bugs are masking the result. The only remaining problem is that the analyzer can't find edits strong enough to clear the gate. That's a search problem, not a math problem.
The Silver Lining
The math killed every promotion. That's the most valuable thing it did.
v0.1.0's p=0.23 was confusing. Was the gate too strict? Was the task set too small? Was the analyzer too weak? We couldn't tell. The math gave us an answer but not a diagnosis.
v0.2.0's math gave us the diagnosis. With 40 tasks and Mistral 24B producing +0.025 mean delta at p~0.5, the numbers are unambiguous: the analyzer finds real but tiny improvements. The gate correctly rejects them. The task set is big enough. The pipeline is correct. The search is the bottleneck.
We now know the exact mathematical relationship between corpus size, movable tasks, and the p-value floor. We know that 40 tasks with 5 movable pairs can clear p<0.05 if the edit is flawless. We know that Mistral moves 1-2 tasks, not 5. We know the gap between "what the analyzer produces" and "what the gate requires" is 3-8 movable tasks.
That gap is measurable. It's tractable. It's the kind of problem that disappears with a better search algorithm — not a bigger model, not a bigger corpus, not a lower threshold. The math told us exactly where the bottleneck is and exactly what it would take to fix it.
The permutation test did not kill our promotion. It saved us from shipping noise as improvement. That is the gate's job, and it did it exactly right.
Links
- GitHub: github.com/deghosal-2026/agent-self-edit
- v0.2.0 release: github.com/deghosal-2026/agent-self-edit/releases/tag/v0.2.0
- v0.2.0 field test report: FIELD_TEST_REPORT.md
- A/B test engine: ab_test.py
- Gate source: gate.py
Top comments (3)
The sign-test floor point is the one that deserves more attention than it usually gets: with 5 discordant pairs out of 26, even a perfect edit can't clear p<0.05, which means the gate was never really evaluating the edit — it was evaluating your corpus size. That's a subtle trap because it looks like the model is failing when actually the experiment is underpowered by construction. The uncomfortable implication is that "expand the corpus" and "the deltas got smaller" can both be true at once: broadening the task set dilutes the fraction of movable tasks, so statistical power and effect size fight each other. One thing I'd try is stratifying — run the significance test only on the subset of tasks the edit could plausibly touch, rather than the full 40, so you're not paying a power tax for tasks where prompt A and B are always going to agree. You still need an honest correction for having chosen the subset, but it stops immovable tasks from drowning real signal. Genuinely one of the more clear-eyed writeups I've seen on why "the agent found improvements" and "the improvements are promotable" are different claims.
Max, the stratification suggestion is genuinely useful and I think it's the right direction for v0.3.0+ for a specific reason the article doesn't quite get to.
The article correctly identifies that expanding from 26 to 40 tasks diluted the effect size (11.5% → 2.5%) because the analyzer's edits still only moved 1-3 tasks regardless of corpus size. But the conclusion it draws — "the fix is better search, not more data" — and the stratification approach aren't in conflict. They address different bottlenecks: better search finds stronger edits, stratification stops irrelevant tasks from taxing the power budget while you're looking.
The practical version I'm considering: when the analyzer produces an edit targeting section X of the prompt, the A/B test could automatically narrow to tasks whose failure mode implicates section X. The analyzer already outputs section and evidence_traces in its JSON — so the task subset is derivable from what the analyzer tells us it's trying to fix.
The honest correction for subset selection is the harder part. Stratifying after seeing the edit inflates the type I error unless you penalize the look. I'd want to pre-declare the task-to-section mapping before any run starts, so the subset is an experimental design choice rather than a post-hoc selection. The corpus already has benchmark roles — extending that to section-affinity tags is tractable.
On your broader point — "improvements" vs. "promotable improvements" being different claims — I think that's the framing I should have led with in the article rather than burying it in the analysis. The article calls it out but the TL;DR is about the math, not the distinction between directional movement and deployed improvement. If I were rewriting the opening today, I'd make that the thesis rather than the subtext.
This is a great example of why a strong evaluation gate matters. The most valuable result isn’t that the edits failed—it’s that the experiments isolated why they failed. Once you know the pipeline and statistical test are working correctly, throwing more data or a bigger model at the problem can become a distraction. The positive-control point is especially important: proving that a genuinely strong edit can clear the gate would give a much clearer baseline for evaluating the search process.