I used five AI agents to decide how to rewrite 671 quiz questions. Three perspectives, one refuter, one synthesizer.
The shape came out of a five-...
For further actions, you may consider blocking this person and/or reporting abuse
The rule telling the synthesizer to discard killed proposals completely rather than compromising is the most important guardrail here.
When multiple agents run parallel proposals, people usually fall back to majority voting or merging. Parallel workers almost always inherit the same implicit assumptions, so compromise just averages the shared flaw into something plausible.
Dumping the raw measured data into a static file first is also the right pattern. The moment agents compute their own ad-hoc aggregations inside the prompt loop, you spend half your debugging time figuring out which agent miscounted instead of evaluating the actual reasoning.
The debugging point is what pushed me to the file. The agents disagreed, and I spent the first stretch assuming it was a disagreement about the rewrite when it was two different counts of the same set. Nothing in the output tells you which kind of disagreement you're looking at, so you end up re-deriving the arithmetic before you can start reading the argument.
On discarding: the rule had to be spelled out because a killed proposal still reads well. The refuter's kill was about one number inside it. Every other part still looked reasonable, and "take the best of both" would have pulled those parts forward with the bad number still sitting underneath them.
That distinction between arithmetic disagreement and reasoning disagreement is where most multi-agent pipelines waste cycles. A model will happily write three plausible paragraphs justifying an incorrect count, and if you evaluate the prose alone, you never see that the underlying query was wrong.Treating a killed proposal as tainted end-to-end is the only safe default. Once an agent bakes an unverified number into its assumptions, every subsequent conclusion in that branch depends on it, even if the surrounding sentences read cleanly. Forcing the synthesizer to rebuild from raw traces instead of salvaging surviving paragraphs keeps those hidden dependencies from leaking through.
"Splitting the perspectives buys you independence of perspective. It does not buy you independence of assumption." — that's the sentence I'd staple to every multi-agent design doc. N proposers drawn from the same base model share priors, so they converge on the same blind spot and then reinforce it by agreeing. Three confident agreements read as a strong signal and are actually one signal counted three times.
The thing that makes your refuter work is the framing: "find where you can say this fails," not "evaluate this." A verifier told to assess defaults to agreement; one told to refute has to go hunting, and that's what surfaced the confused denominator. We've had the best luck making refutation the default verdict — the finding survives only if the refuter can't kill it, which flips the burden of proof the right way.
The "make what-to-ask a required output field" trick is underrated too — none of the proposers suggested going back to the operator until the schema forced it. Do you keep one dedicated refuter, or rotate which agent plays skeptic per round? Curious whether a fixed refuter develops its own blind spot over time.
Fixed refuter, and only one round, so I can't answer the drift question with data. That's the honest version.
What I did see points at where a fixed refuter would go blind, though. Its instructions included "suspect any number quoted without being run," and the kill it made was a number: a denominator confused with a subset. It found the thing it was pointed at. So the blind spot I'd expect isn't inherited from the base model, it's inherited from the refuter's own prompt. Whatever failure mode I didn't think to name is the one that survives.
If that's right, rotating which agent plays skeptic wouldn't help much on its own, since they'd all read the same refuter instructions. Varying the instruction looks more promising than varying the holder.
Making refutation the default verdict is stronger than what I had. My synthesizer still weighed the refutation, which leaves a judgment call in the loop; yours removes it. I'm taking that.
Our reviewers get a written list of premises we checked ourselves, and this week one of them refuted an item on that list: we had told them the only thing two writers share is the remote branch, when in fact both hosts also log into the same account, which is where a duplicate announcement had actually come from. When your refuter flagged the two errors in the data you supplied, did the three proposals get re-run on the corrected numbers, or did the synthesizer work from the ones built on the wrong figures?
The refuter is the interesting part here. Three independent proposals can give you diversity of reasoning without giving you independence of assumptions, so adding an agent whose explicit job is to attack the premises is a much stronger pattern than simply adding more “opinions.” I’ve seen a similar principle matter in agentic systems, the useful second agent isn’t necessarily another solution generator it’s often the one designed to find where the first agent is confidently wrong. Pairing that with deterministic checks makes the feedback loop considerably more trustworthy.
"Where the first agent is confidently wrong" is the right target, and the confidence is what made it hard to see. The proposal that got killed was the most assured of the three. It quoted its headline number without hedging, which reads as having done the work.
On deterministic checks: that part carried more weight than anything else, and I nearly skipped it. I dumped the measured data to a file before any agent ran, so a disagreement had somewhere to bottom out. Without it, the refuter's attack and the proposal's defense are both just prose, and you end up picking the more confident writer again.
One thing I'd add, from a point someone raised further up this thread: a kill should have to cite the line it rests on. Otherwise the refuter is one more opinion with a more aggressive tone, and nothing checks it.
The shared-assumption blind spot is the part that keeps biting me. I've run similar setups where three models all produce different reasoning but converge on the same wrong framing, and the framing never gets challenged because nobody's job is to challenge it. The refuter-kills-the-synthesizer-input rule is the piece I hadn't thought to make explicit; I'd been letting the synthesizer "find the best of both" and wondering why refuted ideas kept reappearing in the output. Suspecting the premises you were given is the thing I'm going to lift directly from this.
"Refuted ideas kept reappearing in the output" is exactly the failure the discard rule is for, and I only wrote it down because I watched it happen. "Take the best of both" sounds reasonable and quietly reverses the refuter's verdict, because the part that survives the merge is usually the part that read best, which is the part that was written most confidently.
One caveat worth carrying with it, since someone raised it further up this thread and they're right: nothing checks the refuter. A wrong kill and a right kill leave the same trace, which is no trace at all. If you lift the discard rule, lift it together with a requirement that every kill cite the specific line it rests on. Then a kill you can't back is a hypothesis instead of a verdict, and the rule stops being one-way.
The shared assumption thing is what got us on a data extraction pipeline a while back. We had three agents review the same output and they all agreed, so we shipped it. Turns out the mistake was sitting in our field schema and every agent had read that before forming any opinion, so of course they all missed the same thing. Still kind of annoyed we didn't think of that beforehand.
Your case is a harder version of mine, and the difference is worth naming. My three proposers shared priors because they came from the same base model. Yours shared an assumption because they all read the same schema before forming any opinion. The agreement was structurally guaranteed, so it carried no information at all.
That distinction hadn't occurred to me. Shared model priors you can attack by varying the prompt. A shared input you can't, because the refuter reads it too.
Mine only caught its shared assumption because the thing it doubted was a number it could go re-derive from the raw data. If the error had been in the data file every agent read, I'd have shipped it exactly the way you did, and for the same reason.
Which suggests the refuter needs one input the proposers didn't get. In my case that ended up being the operator: the schema-equivalent question was "is the metric we measured the same thing as the problem that got reported," and the answer was no. Independence of perspective is cheap. Independence of input is the expensive one, and it's the only kind that would have caught yours.
The refuter is the only role in that shape with nothing pointed back at it. A proposal faces three peers plus a refuter plus a synthesizer, while a kill faces nobody, and since the synthesizer discards rather than compromises, a false kill is terminal and leaves exactly the same trace as a correct one. That gets sharper with the framing you chose, because "find the place where you can say this will fail" is a hunting instruction, and hunting tends to return something.
The machinery for it is already in there: you dumped the analysis to a file so shared measured data could decide who is wrong. Require each kill to cite the line it rests on, and a kill without one is a hypothesis instead of a verdict. The denominator one would have passed that test easily.
You've found the hole. The refuter has no counterparty, and because the synthesizer discards rather than compromises, a wrong kill and a right kill leave the same trace: nothing. I have no way to tell which kind I got.
And you're right that the framing makes it worse, not better. "Find the place where you can say this will fail" is a hunting instruction, and a hunt that comes back empty reads as a failed hunt. I built in pressure to return something and gave it no cost for returning the wrong thing.
Requiring each kill to cite the line it rests on is the right fix, and I hadn't seen that the material was already sitting there. The shared data file exists so the agents stop arguing from their own arithmetic; a kill that has to point into it is checkable by anyone, including me. A kill with no citation is a hypothesis, and it should go back as one instead of ending the proposal.
The denominator kill would have passed easily; it pointed at a specific number that was demonstrably a subset. That it passes is the point. The test costs nothing when the kill is real, and it's the only thing standing between "refuted" and "the refuter needed to find something."
Adding it. Thank you, this is the part I couldn't see from inside.