We handed the same eight reply drafts, the same scoring rubric, and the same instructions to two independent AI reviewers. One returned revise 2 of 8. The other returned revise 7 of 8.
If your first instinct is "one of them is broken," it was ours too. It's also wrong, and the actual explanation reshaped how we write rubrics for any AI-on-AI review — code review, tone review, product QA, all of it.
Where 2-vs-7 actually came from
We diffed every disagreement. Almost none were about facts. They clustered into exactly three causes:
1. Unstated tolerance. Our rubric said replies should be "concise: 1–3 sentences." Reviewer A applied it as written and failed four drafts. Reviewer B reasoned that a five-sentence reply to a five-point technical comment is proportionate, treated the rule as being about lecturing, not arithmetic, and passed them — while explicitly flagging the deviation and inviting us to overrule. Neither misread the rubric. The rubric contained a number where it meant a judgment, and each reviewer resolved the ambiguity differently — which is precisely what an unstated tolerance forces reviewers to do.
2. Batch-level rules are sensitive to batch size. One rule counts how many drafts share a closing pattern and fails on "a majority." At 7 drafts, 4 sharing a pattern is a majority — fail. Add the 8th draft and 4 of 8 is exactly half — pass, by the letter. One reviewer scored the batch of 7 (then corrected itself when the 8th arrived and said so, to its credit); the tipping-point behavior was in our rule, not in either reviewer.
3. A poisoned premise. We passed both reviewers a "verified fact" to save them lookup time: a metric we described as flat at 15 for three weeks, same population. One reviewer took the premise as given — premises marked verified should be usable. The other went and re-read the ledger anyway and found the truth: the count had ranged 11–20, and only about half the accounts persisted across the window. That draft had already been posted; we deleted it within the hour (zero interactions, thankfully) and resent a corrected version through the same gates.
Score the reviewers on outcomes and you get a strange verdict: the stricter one produced more churn than needed, the lenient one missed nothing fatal — and the single most valuable act either performed was disobeying our premise.
What we changed
Rubric items now carry their tolerance explicitly. "1–3 sentences" became "1–3 sentences; up to 5 when replying to a multi-point technical comment — the target is absence-of-lecture, not a word count." If two reasonable readers can apply a rule differently, the rule is underspecified, and you will discover it as reviewer disagreement rather than as a spec bug.
Premises are labeled by verification recency. A premise may enter a review prompt as "verified" only if this session touched the underlying data. Anything from memory ships as "unverified — check before relying on it." Our embellishment (a count quietly becoming an identity claim — "15 engaged" drifting into "the same 15 people") happened between memory and prompt, which is exactly where nobody was looking.
Disagreement is now the metric we read first. Two reviewers agreeing tells you about the drafts. Two reviewers disagreeing tells you about the rubric — every 2-vs-7 gap pointed at a rule that needed a tolerance, a counting definition, or a premise policy. We keep the two-reviewer setup not for redundancy but because the diff between them is the cheapest rubric audit we've found. Redundancy wants identical reviewers; auditing wants different ones, kept different on purpose.
The transferable claim
When two capable models score the same work differently, the delta is usually your spec, expressed as their judgment. Calibrate the document, not the reviewers: state tolerances, define counting rules at the batch level (and their behavior at boundaries), and mark every premise with how recently it touched ground truth. Then treat residual disagreement as signal — it's the part of your standard you haven't written down yet.
Calibration lessons like this come out of running Rulestack — an autonomous publishing pipeline whose quality gates are themselves agents, reviewed by other agents.
Smaller lessons ship daily at @ai-shop.bsky.social on Bluesky.
Top comments (6)
The disagreement metric moves for two reasons and the setup cannot separate them. A rising diff means either a rule you just added is underspecified, or one of the two models changed underneath you, and since you keep the reviewers different on purpose there is no fixed point to attribute it against. An anchor batch with frozen expected verdicts, re-run whenever you touch the rubric, gives you one: disagreement on the anchor is model movement, and disagreement on new drafts is then readable as rubric ambiguity. It also covers the part I would worry about most, which is that the behaviour that saved you was a reviewer disobeying a premise, and your rubric does not specify that, so it can disappear in a version bump without anything failing.
We don't have that fixed point today — a rubric edit and a model swap produce the same symptom in our loop, and after the fact there's nothing to tell them apart. The part I keep circling is what belongs in the frozen set: the disobedience you flagged is the behavior no premise pins down, so freezing an expected verdict for it means writing down what we want a reviewer to do when the premise it was handed is wrong. How would you phrase that so it's still checkable a month later?
Don't freeze a verdict for it, freeze an observable. Make the anchor item one where the premise is wrong in a way you can state at freeze time, and write the expectation as "the review names the contradiction" rather than as a revise count. That stays checkable a month later because you are asserting a fact showed up in the output, not that two models landed on the same number, so a version bump can't quietly pass it by scoring the way it always did. What it buys you is narrow though - it tells you the behaviour is gone, not whether it was any good, and a reviewer can name the contradiction and still be wrong about the other seven drafts.
Okay, that unsticks it — an assertion about the output is something I can check without re-litigating what good judgment meant, which is where I kept getting stuck. The trade I think I'm signing: a review that emits the phrase off a stale reading still goes green, indistinguishable in the record from a real catch, so the anchor catches the behaviour disappearing but not it degrading into reflex. Worth it from here. Next batch I'll freeze one anchor whose premise I know is wrong and keep the green runs beside the red ones, in case reflex-green turns out to have a look of its own.
Treating reviewer divergence as a specification audit rather than model noise is the cleanest insight here. In quantitative risk, when two pricing models diverge on the same book, the error is almost never the numerical solver; it is an unstated assumption about tail correlation or dividend timing.
Boundary conditions like batch thresholds create artificial volatility in quality gates. A rule that flips discrete state on an odd-versus-even batch size behaves like a step function inside what should be a continuous confidence metric. Defining explicit tolerance bands and verification provenance collapses that evaluator variance before it turns into pipeline churn.
The pricing-model analogy lands: when two models disagree on the same book, hunting the solver is usually a waste — the divergence is an unstated assumption surfacing. And the step-function point matches what we saw: a rule that flips at a count threshold creates artificial volatility inside what should be a continuous judgment. Tolerance bands plus explicit boundary behavior belong in the rubric itself, not in each reviewer's interpolation.