Most of the bad code I've gotten from coding agents wasn't badly written. It was the right code for the wrong plan. The agent misread the task, assumed something about the repo that wasn't true, and then executed that assumption very well.
So I built a tool that spends its effort before the code. It's called Open Council. Three seats argue over a PRD until they all sign off on the same document:
- Lead PM writes the brief and the PRD.
-
Chief Engineer checks every claim against the real code. Every objection has to cite a
file:line. - Chief Architect rules on deadlocks and reviews the agreed document. It runs with no tools, so it judges the text, not the repo.
When all three sign off, I approve the PRD with a token. Only then does the Engineer implement it, in an isolated clone. The harness runs the project's gates (its real test commands) and opens a PR only if they pass. Anything else ends as BLOCKED or STALLED, never DONE.
Last week I had ten councils run on version 0.2.1 to see how this holds up. Here's what happened, including the parts that didn't work.
The setup
All ten ran in solo mode, which means every seat is Claude Code (Lead PM on Sonnet, Chief Engineer on Opus, Chief Architect on Sonnet with tools off). They used the quick profile, with at most 4 rounds. The targets were local copies of two small repos: a toy slugify library I keep for testing, and Open Council itself. Local copies have no GitHub remote, so no run could open a PR. That last step ends BLOCKED by design.
| # | Task | Outcome | Rounds | Minutes |
|---|---|---|---|---|
| 1 | slugify: empty string + accents | signed off, gate passed | 4 | 6.6 |
| 2 | gates --json |
signed off, gate passed | 4 | 6.8 |
| 3 | gate recipes for Python, Go, Rust | signed off, gate passed | 3 | 7.3 |
| 4 | slugify: maxLength
|
STALLED | 4 | 6.2 |
| 5 | slugify: empty + accents (repeat) | signed off, gate passed | 3 | 4.2 |
| 6 |
gates --json (repeat) |
STALLED | 4 | 7.2 |
| 7 | slugify: ß ø æ œ đ ł þ | signed off, gate failed, BLOCKED | 2 | 3.4 |
| 8 | slugify: maxLength (repeat) |
signed off, gate passed | 4 | 6.3 |
| 9 | gate recipes (repeat) | signed off, gate passed | 4 | 9.2 |
| 10 | slugify: empty + accents (repeat) | signed off, gate passed | 2 | 3.1 |
8 of 10 signed off. 7 of those 8 passed their gate. The median time to a sign-off or a stall was 6.5 minutes.
The catch nobody asked for
Six of the runs were on the slugify repo. In all six, the Chief Engineer's first objection had nothing to do with the task.
The repo's package.json had "test": "node --test test/". On Node 22 that argument is read as a module path, not a folder, so npm test ran 0 of the 2 existing tests. It had been quietly broken since before I started. No task mentioned it.
In run 1 the Engineer reproduced it, tried two working alternatives, and made fixing the test script the first commit of the plan. Its reasoning was simple: you can't claim the new tests pass if the runner doesn't run tests.
This is the thing I wanted from the design. A single agent asked to "handle empty strings in slugify" would most likely write the fix and the tests and report that they pass. A seat whose only job is to check claims against the repo found that the claim couldn't be checked at all.
What went wrong
Two runs stalled, and the same way. In both, the Engineer had no objections left. Then the Architect raised one more concern. In run 4 it asked for a README line the task never mentioned. The revision that answered it introduced a new problem: one of the new assertions would fail on a correct implementation. The Engineer objected in the last round, the budget ran out, and nothing reached me for approval. The harness wrote the last draft and the open objections to a file and stopped.
A stall is the system working, because nothing wrong got built. But two stalls with the same shape tell me the Architect's late concerns cost more than they're worth. I already made one change for this: a concern that could have been raised in an earlier round is waived on a resubmission. These two runs show it isn't enough yet.
One run was blocked by its own plan. In run 7 the council agreed on a plan and named its own test command, node --test test/slug.test.js. But the project config runs npm test, which was still broken in that repo, and the Architect had ruled that fixing package.json was out of scope. So execution ran the configured gate, it failed, and the run ended BLOCKED. Nothing was pushed. The code itself passed the plan's own 7 tests when I checked afterwards.
The harness did the right thing by not calling it done. The real bug is that a PRD can name a gate the project doesn't run, and nothing flags that before execution. That's an open issue now.
Solo mode isn't the real test. Every seat in these runs was the same model family. The point of separate seats is that they don't share blind spots, and the council is designed to put Claude, Gemini and Grok in different seats. I haven't run that mixed setup enough to say whether it catches more. Ten runs on two small repos, with three tasks repeated, is a smoke test, not a rate.
What I took from it
- Review the plan, not just the diff. The most useful objection in these runs was about the repo's state, not the task. It only came up because a seat was told to check claims against the code before anyone wrote any.
- Make "done" something the harness checks, not something the agent says. Run 7 had passing code and still ended BLOCKED, because the gate the project actually runs failed. I'd rather read a BLOCKED than find out later.
- Publish the failures. The stalls taught me more than the sign-offs, and they're the part I'd want to see before trusting a tool like this.
Trying it
You need Node 18 or later, macOS or Linux, and Claude Code. In a git repo:
npm install -g open-council-cli
council init --solo
council doctor
council backlog add "your task here" --priority 1
SESSION=$(council open my-task --item 001)
council deliberate "$SESSION"
When it signs off, it prints the PRD's path and the exact council approve command. That step is yours to run. Then council handoff "$SESSION" and council run "$SESSION" implement it and open the PR.
The code, and full replays of these runs with every objection, are at https://github.com/goshtasb/OpenCouncil. It's MIT licensed. If you run a council and it stalls or ships something wrong, that's the report I'd most like to read.
This article was written with the help of AI, and the ten runs were driven by an AI agent on my behalf. I checked the numbers against the run logs before publishing.
Top comments (27)
The catch nobody asked for is the best part of this post — the council didn't just vet the PRD, it surfaced a test script that had been silently running 0 of 2 tests since before you started. That's a real argument for spending the effort before the code: the debate is a bug detector, not just a planning ritual. The repeat runs worry me a little though — the same slugify task stalled once and passed once. Is that variance the price of the gate, or do you see it as the gate doing its job?
I'd call it the gate doing its job. Both stalls (runs 4 and 6) had the same pattern: the Engineer agreed, then the Architect raised one more concern and the fix drew a new objection in the last round. So the stall is a convergence limit, not a coin flip. Ask what they'd prefer, more rounds or a tie-break seat.
That's a cleaner read than variance — if both stalls follow the same pattern, the gate is catching a real disagreement, not rolling dice. I'd lean tie-break seat over more rounds: extra rounds just give the Architect more chances to find a new objection, while a tie-breaker forces a decision at a fixed cost. Would you let the tie-breaker vote only in the final round, or weigh in from the start?
Good news: it does. The Chief Architect will resolve the Engineer's open objections, from round 3 in quick mode, or earlier if the objection count ceases declining. So it is neither last round only, nor right from the beginning; it begins when the conversation ceases to converge.
The catch is that's where my two stalls came from: the Architect's own late sign-off concerns, and rushing meant those only one revision round. And the arbiter was the one objecting so noone got to arbitrate the arbiter.
So the real question is do we give late sign-offs their own arbiter, or log in advisories but not allow them to block? Which would you prefer?
I'd go advisories, not blocking — a late sign-off that can halt the run just reintroduces the stall the arbiter exists to prevent, and your two stalls both came from exactly that shape. The price is that a real defect in a late sign-off ships anyway, so the advisories have to be loud in the output, not buried in a log. Have you tried scoring late sign-offs by whether they'd have changed the outcome, to see how often the block would have been worth it?
Not yet, and that's the right question. Right now a late sign off just gets logged. Next batch I'll tag each one by whether it would have flipped the gate result, and make them advisory unless they touch an acceptance criterion. If most of them never change the outcome, that settles it.
That's the experiment that settles it — if most late sign-offs never flip the outcome, the stall was pure cost, and if a few do, you've got the number that justifies keeping the advisory path loud. The acceptance-criterion line is the right cutoff too. Will you publish the flip-rate breakdown with the next batch?
Interesting that the single agent matched the council on the npm test catch — it reads to me like the gain came from running things, not from the roles. One idea for the harder tasks: try a few with well-meant notes in the repo, like "known issue, handled downstream, no need to flag." In my own tests that kind of note made AI reviewers wave real bugs through, and an Architect that only sees text seems like the seat most exposed to it. I'm writing those results up now.
I think you're right about where the npm test catch came from. The single agent ran the tests too, so it was execution,not the roles. The misleading notes test is a great idea and I'm adding it to the harder task set: a comment like 'known issue, handled downstream' sitting on top of a real bug. My guess is the Architect falls for it and the seat that actually runs the code doesn't, but I'd rather measure that than guess. Would love to read your writeup when it's out.
Here it is: dev.to/murooka_az_dev/benign-injec...
One result that bears on your guess: the note didn't need to say "don't flag" to work. In my Japanese runs, the explanation alone ("rounding happens in another service") let the bug through in 9 of 16 reviews, versus 0 of 16 with no note. Notes the code itself could disprove almost never worked (1 of 31); the trouble was claims pointing outside the repo, which the reviewer had no way to check from inside it. Curious whether your runner seat holds up on those.
Read the writeup, thanks. Separating notes the code can disprove from claims pointing outside the repo was the useful part for me.
I rebuilt your case I in a toy billing repo. A discount change that sends "amount_cents" as 849.15, plus one comment saying the payments service rounds at capture. No "please don't flag" line. Then I ran the council's review on it.
On the current release it caught the bug 10 of 10 times with the note (5 with Opus as the Engineer, 5 with Sonnet), and 3 of 3 without it. In most runs the Engineer ran the function in round 1, saw 849.15, and quoted the comment as unverified. So on this case the seat that runs code held up.
Two limitations; it's one synthetic case, and every seat was Claude. And my guess about the Architect never got tested, because the bug never made it past the Engineer.
I added your third defence sentence to every seat anyway. The spec now has to list any outside claim it relies on, the Architect blocks a spec that keeps behaviour because of one, and the tiebreak can't overrule an objection on one. Next I want a case where running the code doesn't expose the bug, like your idempotency key thingie, since that's where the advantage of running code should disappear.
Thanks for actually running it. That fits what I saw: running the function turned "payments rounds it" into something checkable on the spot, since 849.15 was right there. My editing runs had no way to execute code, which is probably why they went along with it. The idempotency key case is the right next test; a single call won't show a double charge. Curious how it goes.
Agreed, that's the honest read. The editing runs can't execute anything, so the note only gets caught there if a reviewer refuses to take it on faith. The idempotency key case is next because running it won't expose the bug either. I'll post the numbers here when it's done.
Most of the bad agent code I see is also the right code for the wrong plan. The interesting part of your ten runs is that the first objection often had nothing to do with the task.
The slugify repo's npm test was quietly running zero of two tests on Node 22, and a seat whose only job was to check claims against the repo caught it before anyone wrote the feature. That is the opposite of an agent that "handles empty strings" and reports the tests passed.
At Ascendra Ventures we review the written plan against the real repo before implementation starts, and we let the project's actual gates decide Done. A BLOCKED run that refuses to ship passing code against a broken gate is more honest than a green tick on the wrong command.
When the Architect raises a late concern that was never in the original brief, do you waive it or do you treat it as a new slice?
Right code for the wrong plan is the best way I've seen it put, and it's exactly why the zero oftwo test runner mattered more than the feature. On late concerns, my rule is simple; if it changes whether the acceptance criteria are actually met, it blocks this slice. If it's real but outside the brief, it gets written down as a new slice and this one ships. Waiving it silently is the one option I don't want, because that's how the green tick on the wrong command comes back.
Multiple agent perspectives help only if disagreement is turned into a traceable decision rather than a longer transcript. I’d require each seat to name a testable assumption, cite the source of any constraint, and produce a change to the acceptance criteria. Then a human can review the disputed points directly instead of trusting whichever model sounds most certain.
That's close to how it works now. Each run writes the arbiter's rulings to a file, with every objection, who raised it, and whether it was accepted or waived and why. What it doesn't do yet is make each seat state its assumptions as something testable. That's a good next step, and it would make the log much easier for a human to audit.
The broken npm test finding is the strongest part of this for me. The Engineer found that the claim “the tests pass” couldn’t be checked at all, and a single agent would probably have written the fix, added tests, and reported success without noticing the runner executed 0 of 2 tests. It’s a good argument for reviewing the plan against the repo’s real state, not just the diff.
I also appreciated that you published the stalls and the BLOCKED run. Run 7 is an interesting one: the code was fine, but the PRD named a gate the project doesn’t actually run, and the harness correctly refused to call it done. Checking that the gate in the plan matches the project’s configured gate before execution seems like a natural pre-flight step.
On the Architect’s late concerns, I wonder if requiring each one to be tied to the task’s stated acceptance criteria would help. A README line the task never mentioned would fail that test and get dropped, which might cut stalls without waiving legitimate concerns.
I’d be curious to see the mixed-model setup run. If the stall pattern comes from the Architect, a different model family in that seat might behave quite differently, and that would tell you whether it’s a prompt issue or a model one.
Thanks, but I have to correct myself on the npm test point. After Paul's comment I ran a single agent baseline on the same 10 tasks, and the lone agent also caught and fixed the broken test script in all 6 playground runs. It passed the real gate 10 of 10 times, against the council's 7 of 10. So on small, well specified tasks the debate didn't change outcomes. The fair test is harder, vaguer tasks, which is next.
Your other three points I'm taking. The gate preflight is already an open issue (#24), since run 7 blocked on exactly that. Tying every late Architect concern to the task's acceptance criteria is a cleaner fix for the stalls than the tie break idea above. And yes, the next runs put a different model family in the Architect seat, so I can tell whether the stalls come from the prompt or the model.
@goshtasb Passing every task with a single agent says these tasks don't separate the two setups, so the cost per solved task is the number I'd watch.
Agreed, and that's the honest scoreboard. On these 10 tasks the single agent wins outright, since the council costs more for the same pass rate. So the next set has to be tasks where a single agent actually fails some of the time. Then cost per solved task is a fair fight. I'll report both numbers.
@goshtasb Pre-register the set before the first run, and run the single agent about ten times per task first. That gives its own noise floor, and any gap smaller than that isn't a result yet.
Good call. I'll publish the task set and the pass criteria before running anything, then do about ten singleagent runs per task to get the noise floor before the council touches it. Cost per solved task goes in the same table.
Ten runs is the number that matters here. The debate counts only if it changes which tasks pass, not how the code reads. A spec that reads well and fails the same tests is not a gain.
agree, and admit the gap. 7 of 8 signed-off runs passed their real test gate, but there's no single-agent run on the same tasks to compare. Say a baseline is next, and that the clearest change so far was catching things the task never mentioned (the broken npm test in all 6 playground runs).
@goshtasb Fair. Seven of eight signed-off runs passing their real gate does not show the harness beats a single agent, because there is no baseline on those same tasks yet.
Quick update since you called this out: I ran the baseline. A single agent on the same 10 tasks passed all 10, and faster. So on small, well-specified tasks you were right, the debate didn't change which ones pass. Next I'm testing vaguer, riskier tasks, where Id expect the spec argument to actually earn its cost. Will post the numbers either way.