I used five AI agents to decide how to rewrite 671 quiz questions. Three perspectives, one refuter, one synthesizer.
The shape came out of a five-agent cap, but nothing in it turned out to be waste. Generating independent proposals is not enough. You need something whose job is to kill them.
The refuter killed one of the three
One proposal went down for this:
It never ran the verification it most strongly demanded, on its own headline number — it had confused the denominator with a subset.
Three proposals side by side did not surface that. I didn't catch it either, reading them myself.
The refuter's instructions were explicit:
- Do not read this to agree with it
- Find the place where you can say "this will fail in execution"
- Suspect any number quoted without being run
The real win: the assumption all three shared
All three silently assumed that the metric they measured was the same thing as the problem that had been reported. Only the refuter went at that.
When I actually asked the operator, the answer was two different things entirely. Fixing only the headline metric would have missed the target.
Splitting the perspectives buys you independence of perspective. It does not buy you independence of assumption.
Don't let the synthesizer compromise
The synthesizer was told plainly: discard anything the refuter killed.
Allow compromise and a dead proposal comes back diluted. "Take the best of both" does not un-refute the reasoning that killed one of them.
Hand everyone the same measured data, up front
Let each agent do its own aggregation and they recount the same thing repeatedly and disagree on the results. I dumped the analysis to a file first and had all of them read it.
On top of that: write only numbers you produced by running something. No estimates.
One agent still got its own recount wrong. Because the report was fully reproducible, the refuter could determine that the agent was the thing that was off. Shared measured data decides who is wrong, not just that something is.
It also found errors in what I supplied
The refuter flagged two errors in the measured data I had handed them. Both were real.
Instruct it to suspect the premises it was given, and it catches your mistakes too.
Make "what to ask" a required output field
I required the synthesizer to emit open_question — things the operator should decide.
What came back: ask the operator which items they actually found confusing; ten minutes settles the direction. That pays for itself before you gamble 4 to 70 hours.
All three proposals started from one remark by the operator, and not one of them proposed going back and asking. It doesn't appear unless you require the field.
I hand-copied the work split and got a quarter of it wrong
This one is mine.
Distributing 116 sites across five agents, I hand-copied the script's assignment output into JSON. I got 28 of them — a quarter — wrong. Names that didn't exist went out; 28 real ones never did.
I caught it by diffing against the real data right after launch, before a single edit landed.
Three fixes:
- Diff what you handed out against what generated it, as the first step after launch
- Verify names down to file existence before handing them over (added an
assert) - Remove the transcription step entirely. The script now emits a string you paste as-is
The cause wasn't carelessness. It was that a copying step existed at all.
Fix the standard in a document before distributing
Before handing out 390 rewrites, I did 76 myself and wrote the pattern into a document — good examples, bad examples, and what must not be touched.
The thing that breaks first when you split work is consistency of standard. One dialect per agent is worse than no standard at all.
Machine checks are the gate; the writing standard travels as prose. I also stated: do not loosen the check to pass. If it feels too strict, report it — don't edit it. Never leave the gate adjustable from the inside.
If everyone falls in the same hole, it's your handoff
Six agents, one video cut each: every one of them hit "the font file isn't here," and solved it four different ways — bare font names, embedding a font from another project, fetching from the network.
I only found out because each report was required to include "where the instructions and reality diverged."
When all of them raise the same point, it isn't the agents. It's a gap in what you handed them.
The next round, I folded the previous four failures into the instructions. None of the four recurred. But sealing holes reveals the next ones — three agents independently fell into a different one.
Requiring those reports is how you learn what to hand over next time.
Takeaways
- Proposals alone don't find errors. Stand up an independent refuter
- Separate perspectives still share assumptions. Only the refuter attacks those
- Don't let the synthesizer compromise. Killed proposals return diluted
- Distribute the same measured data first. It decides who is wrong
- Tell it to suspect your premises. It will catch your errors
- Require an "open questions" field. It doesn't appear otherwise
- Never build a step that requires transcription. I got a quarter wrong
- Everyone in the same hole means the handoff is missing something
Separately from this, I build a desktop AI agent called Wisp. It stands on your desktop, answers when you talk to it, and runs commands when you ask — always showing you what it's about to do first.
Top comments (14)
The rule telling the synthesizer to discard killed proposals completely rather than compromising is the most important guardrail here.
When multiple agents run parallel proposals, people usually fall back to majority voting or merging. Parallel workers almost always inherit the same implicit assumptions, so compromise just averages the shared flaw into something plausible.
Dumping the raw measured data into a static file first is also the right pattern. The moment agents compute their own ad-hoc aggregations inside the prompt loop, you spend half your debugging time figuring out which agent miscounted instead of evaluating the actual reasoning.
The debugging point is what pushed me to the file. The agents disagreed, and I spent the first stretch assuming it was a disagreement about the rewrite when it was two different counts of the same set. Nothing in the output tells you which kind of disagreement you're looking at, so you end up re-deriving the arithmetic before you can start reading the argument.
On discarding: the rule had to be spelled out because a killed proposal still reads well. The refuter's kill was about one number inside it. Every other part still looked reasonable, and "take the best of both" would have pulled those parts forward with the bad number still sitting underneath them.
That distinction between arithmetic disagreement and reasoning disagreement is where most multi-agent pipelines waste cycles. A model will happily write three plausible paragraphs justifying an incorrect count, and if you evaluate the prose alone, you never see that the underlying query was wrong.Treating a killed proposal as tainted end-to-end is the only safe default. Once an agent bakes an unverified number into its assumptions, every subsequent conclusion in that branch depends on it, even if the surrounding sentences read cleanly. Forcing the synthesizer to rebuild from raw traces instead of salvaging surviving paragraphs keeps those hidden dependencies from leaking through.
"Splitting the perspectives buys you independence of perspective. It does not buy you independence of assumption." — that's the sentence I'd staple to every multi-agent design doc. N proposers drawn from the same base model share priors, so they converge on the same blind spot and then reinforce it by agreeing. Three confident agreements read as a strong signal and are actually one signal counted three times.
The thing that makes your refuter work is the framing: "find where you can say this fails," not "evaluate this." A verifier told to assess defaults to agreement; one told to refute has to go hunting, and that's what surfaced the confused denominator. We've had the best luck making refutation the default verdict — the finding survives only if the refuter can't kill it, which flips the burden of proof the right way.
The "make what-to-ask a required output field" trick is underrated too — none of the proposers suggested going back to the operator until the schema forced it. Do you keep one dedicated refuter, or rotate which agent plays skeptic per round? Curious whether a fixed refuter develops its own blind spot over time.
Fixed refuter, and only one round, so I can't answer the drift question with data. That's the honest version.
What I did see points at where a fixed refuter would go blind, though. Its instructions included "suspect any number quoted without being run," and the kill it made was a number: a denominator confused with a subset. It found the thing it was pointed at. So the blind spot I'd expect isn't inherited from the base model, it's inherited from the refuter's own prompt. Whatever failure mode I didn't think to name is the one that survives.
If that's right, rotating which agent plays skeptic wouldn't help much on its own, since they'd all read the same refuter instructions. Varying the instruction looks more promising than varying the holder.
Making refutation the default verdict is stronger than what I had. My synthesizer still weighed the refutation, which leaves a judgment call in the loop; yours removes it. I'm taking that.
Our reviewers get a written list of premises we checked ourselves, and this week one of them refuted an item on that list: we had told them the only thing two writers share is the remote branch, when in fact both hosts also log into the same account, which is where a duplicate announcement had actually come from. When your refuter flagged the two errors in the data you supplied, did the three proposals get re-run on the corrected numbers, or did the synthesizer work from the ones built on the wrong figures?
The refuter is the interesting part here. Three independent proposals can give you diversity of reasoning without giving you independence of assumptions, so adding an agent whose explicit job is to attack the premises is a much stronger pattern than simply adding more “opinions.” I’ve seen a similar principle matter in agentic systems, the useful second agent isn’t necessarily another solution generator it’s often the one designed to find where the first agent is confidently wrong. Pairing that with deterministic checks makes the feedback loop considerably more trustworthy.
"Where the first agent is confidently wrong" is the right target, and the confidence is what made it hard to see. The proposal that got killed was the most assured of the three. It quoted its headline number without hedging, which reads as having done the work.
On deterministic checks: that part carried more weight than anything else, and I nearly skipped it. I dumped the measured data to a file before any agent ran, so a disagreement had somewhere to bottom out. Without it, the refuter's attack and the proposal's defense are both just prose, and you end up picking the more confident writer again.
One thing I'd add, from a point someone raised further up this thread: a kill should have to cite the line it rests on. Otherwise the refuter is one more opinion with a more aggressive tone, and nothing checks it.
The shared-assumption blind spot is the part that keeps biting me. I've run similar setups where three models all produce different reasoning but converge on the same wrong framing, and the framing never gets challenged because nobody's job is to challenge it. The refuter-kills-the-synthesizer-input rule is the piece I hadn't thought to make explicit; I'd been letting the synthesizer "find the best of both" and wondering why refuted ideas kept reappearing in the output. Suspecting the premises you were given is the thing I'm going to lift directly from this.
"Refuted ideas kept reappearing in the output" is exactly the failure the discard rule is for, and I only wrote it down because I watched it happen. "Take the best of both" sounds reasonable and quietly reverses the refuter's verdict, because the part that survives the merge is usually the part that read best, which is the part that was written most confidently.
One caveat worth carrying with it, since someone raised it further up this thread and they're right: nothing checks the refuter. A wrong kill and a right kill leave the same trace, which is no trace at all. If you lift the discard rule, lift it together with a requirement that every kill cite the specific line it rests on. Then a kill you can't back is a hypothesis instead of a verdict, and the rule stops being one-way.
The shared assumption thing is what got us on a data extraction pipeline a while back. We had three agents review the same output and they all agreed, so we shipped it. Turns out the mistake was sitting in our field schema and every agent had read that before forming any opinion, so of course they all missed the same thing. Still kind of annoyed we didn't think of that beforehand.
Your case is a harder version of mine, and the difference is worth naming. My three proposers shared priors because they came from the same base model. Yours shared an assumption because they all read the same schema before forming any opinion. The agreement was structurally guaranteed, so it carried no information at all.
That distinction hadn't occurred to me. Shared model priors you can attack by varying the prompt. A shared input you can't, because the refuter reads it too.
Mine only caught its shared assumption because the thing it doubted was a number it could go re-derive from the raw data. If the error had been in the data file every agent read, I'd have shipped it exactly the way you did, and for the same reason.
Which suggests the refuter needs one input the proposers didn't get. In my case that ended up being the operator: the schema-equivalent question was "is the metric we measured the same thing as the problem that got reported," and the answer was no. Independence of perspective is cheap. Independence of input is the expensive one, and it's the only kind that would have caught yours.
The refuter is the only role in that shape with nothing pointed back at it. A proposal faces three peers plus a refuter plus a synthesizer, while a kill faces nobody, and since the synthesizer discards rather than compromises, a false kill is terminal and leaves exactly the same trace as a correct one. That gets sharper with the framing you chose, because "find the place where you can say this will fail" is a hunting instruction, and hunting tends to return something.
The machinery for it is already in there: you dumped the analysis to a file so shared measured data could decide who is wrong. Require each kill to cite the line it rests on, and a kill without one is a hypothesis instead of a verdict. The denominator one would have passed that test easily.
You've found the hole. The refuter has no counterparty, and because the synthesizer discards rather than compromises, a wrong kill and a right kill leave the same trace: nothing. I have no way to tell which kind I got.
And you're right that the framing makes it worse, not better. "Find the place where you can say this will fail" is a hunting instruction, and a hunt that comes back empty reads as a failed hunt. I built in pressure to return something and gave it no cost for returning the wrong thing.
Requiring each kill to cite the line it rests on is the right fix, and I hadn't seen that the material was already sitting there. The shared data file exists so the agents stop arguing from their own arithmetic; a kill that has to point into it is checkable by anyone, including me. A kill with no citation is a hypothesis, and it should go back as one instead of ending the proposal.
The denominator kill would have passed easily; it pointed at a specific number that was demonstrably a subset. That it passes is the point. The test costs nothing when the kill is real, and it's the only thing standing between "refuted" and "the refuter needed to find something."
Adding it. Thank you, this is the part I couldn't see from inside.