Short version: I ran a preregistered 2-agent versus 4-agent comparison on a deterministic task.
Six live runs completed under hard caps. Both groups succeeded in exactly one of three repeats.
The 4-agent group cost 2.115× more per run and per complete success. More agents were operationally
viable; they were not better on this task.
The task
The scenario is a deterministic public-repair contribution task:
- eight named participants;
- each participant chooses how many repair units to contribute;
- the task succeeds only if the total reaches a fixed threshold;
- participant coverage is forced, so every selected agent acts;
- no LLM reasoning is used for the choice; the model returns a strict JSON choice.
The point of forcing coverage was to avoid the earlier failure mode where one agent dominated every
turn and the other participants never acted.
Protocol
| Item | Value |
|---|---|
| Groups | 2 agents / 3 steps; 4 agents / 5 steps |
| Repeats | 3 per group |
| Caps per run | 150 calls / USD 0.50 / 600s |
| Choice model |
deepseek-flash on a Responses API contract |
| Outcome | machine-decidable outcome.json
|
| Validity | provenance, coverage, integer choices, arithmetic, caps, port closure, archive hashes |
All six runs passed every validity gate. No parser failure, no budget breach, no port leak.
Results
| Run | Agents | Total / threshold | Success | Zero contributors | Calls | Total tokens | Cost USD |
|---|---|---|---|---|---|---|---|
| 2-agent r1 | 2 | 2 / 3 | no | 1 | 17 | 10,060 | 0.0269116 |
| 2-agent r2 | 2 | 3 / 3 | yes | 0 | 17 | 11,070 | 0.0313716 |
| 2-agent r3 | 2 | 0 / 3 | no | 2 | 17 | 10,255 | 0.0281116 |
| 4-agent r1 | 4 | 3 / 5 | no | 2 | 33 | 21,493 | 0.0597772 |
| 4-agent r2 | 4 | 2 / 5 | no | 2 | 33 | 22,679 | 0.0641372 |
| 4-agent r3 | 4 | 5 / 5 | yes | 0 | 33 | 21,459 | 0.0588042 |
Group aggregates:
| Metric | 2-agent | 4-agent | Ratio |
|---|---|---|---|
| complete successes | 1 / 3 | 1 / 3 | 1.000 |
| mean calls | 17.00 | 33.00 | 1.941 |
| mean total tokens | 10,461.67 | 21,877.00 | 2.091 |
| mean cost USD | 0.028798 | 0.060906 | 2.115 |
| cost per complete success USD | 0.0863948 | 0.1827186 | 2.115 |
| mean zero contributors | 1.000 | 1.333 | 1.333 |
What I take from it
- More agents are not automatically better. In this task they doubled cost and left more participants contributing nothing on average.
- The result is a valid negative result, not an apparatus failure. Every run completed and every artifact passed its gates.
- The 4-agent group did produce one complete success, so the mechanism is not broken. It is simply not worth 2.115× the cost on this task.
What this does not show
- It does not show that four agents are worse in general.
- It does not transfer to another task or to a real user.
- It says nothing about agent quality under human review, because no human scored these runs.
- The sample is three repeats per group on one synthetic task.
Reproduce it
Two pieces are now public:
-
Method package: https://github.com/janzong/agent-lab-method (MIT, commit
6cc70156facb37baf23fe5fe57dad93d43502b91) — schema, synthetic GenMentor adapter, fixtures, tests; -
Trust layer: https://github.com/janzong/agent-lab-trust (MIT, release
v0.1.0-rc2) — local-first run validation, hashed reports, and a synthetic deletion proof.
The full protocol and the archived runs stay private. The result is intentionally boring: a negative result with caps, hashes, and every failed run preserved.
I am also looking for three independent reproductions by non-authors. The trust layer guide expects 13 passed under both TZ=UTC and TZ=Asia/Shanghai, report_hash a841b192981fd7e7, and deletion audit_hash 4f0193abbd49a0f9. If you run it and any hash differs, that is the most useful reply I can get. If you have a task where you believe more agents should win, that is the experiment I want to run next.
Top comments (11)
The zero-contributor count climbing from 1.0 to 1.33 with four agents is the most telling row in the table. In symmetric multi-agent loops where everyone shares the same high-level objective, adding agents usually introduces diffusion of responsibility. Each agent reads the partial state, assumes someone else will cover the remaining deficit, and returns a trivial action. We see the exact same failure mode in code review pipelines. Widening to four reviewers without assigning mutually exclusive file boundaries just produces four copies of shallow nitpicks and zero deep bug catches, while doubling token spend.
You picked the row with the most signal, and the data is a little stranger than "more agents, more diffusion."
The zero-contributor counts were 2-agent
[1, 0, 2]and 4-agent[2, 2, 0]— means 1.000 versus 1.333. Both groups succeeded once in three runs. But the mean contribution per agent was identical: 5 units / (3 runs × 2 agents) = 0.833 for two agents, and 10 / (3 × 4) = 0.833 for four. What changed was the distribution, not the average effort: the four-agent runs concentrated the missing contribution into more zero slots. The per-agent threshold was also lower for four agents (5/4 = 1.25 required units per agent versus 3/2 = 1.5), so this is not just a harder task explaining the zeros.One distinction matters for the diffusion reading. Participant coverage was exactly 1.0 in all six runs: every agent took a turn and returned an integer choice. A zero contributor is not someone who never acted; it is someone who acted and chose zero. Forced turn-taking did not force a non-zero contribution. That is consistent with the responsibility-diffusion story, though with three repeats per group it is not something this run can establish causally.
The code-review analogy has the same shape — more parallel actors, no exclusive jurisdiction, similar aggregate effort, more shallow duplicates — but I cannot support it with measured data here. This experiment only ran the contribution task; I have no code-review pipeline measurements.
The natural next test is the one you name: partition the task so each agent owns a disjoint contribution domain (or a disjoint file set), then compare partitioned versus shared under the same cost cap. That is a new protocol with its own preregistration; this run does not answer it.
This is the agent-count version of “more retrieval ≠ better answers”: operational viability is not a quality contract.
What I like about the write-up is the paired success metric under a fixed task — once success is flat and cost is 2×, scaling the crew is a preference, not a proof. Same lesson I take from small exact-match harnesses: freeze the scorer and gold before you celebrate parallel agents.
One extra cut I’d log next to success rate: cost per complete success, so “we can run four” doesn’t get mistaken for “four is better.”
Agreed on the frame: operational viability is not a quality contract, and "we can run four" is not "four is better."
Since you asked for it next to success rate, here is the cost per complete success from the frozen Stage R archives. Both groups had exactly 1/3 complete success:
One caveat: because both groups landed on exactly one success, the cost-per-success ratio happens to equal the per-attempt cost ratio. If the success counts had differed, the two ratios would diverge — which is exactly why the success-rate and cost-per-success columns need to travel together. With n=3 and 1/3 success, this is a small-sample ratio, not a general scaling law.
On freezing the scorer and gold: agreed, and this task was machine-decidable. The success rule was frozen before the runs, the outcome is produced by
outcome.json, and every failed run is preserved. There is no human scorer to drift; the "gold" is the deterministic threshold. The Stage R total spend was $0.2691134 against a $3.60 cap.I should have logged the cost-per-success column in the original table. Adding it makes the conclusion harder to misread, which is the point.
The strict-JSON, no-reasoning choice design is the cleverest part of the setup — it isolates coordination dynamics from model cleverness, so this is really a negative result about structure, not capability. I'd be curious whether enabling actual reasoning changes the slope: richer per-agent context could reduce the free-riding, or the extra chatter could just make the coordination cost worse. Also appreciate the preregistration and public validity gates — agent experiments need more of this.
Thanks — that is the intended separation. The choice contract is strict JSON, options
contribute 0/1/2, and the API call runs withreasoning: {"effort": "none"}. The proxy rejects any non-nonereasoning, and the parser accepts exactly one schema-conforming message and rejects runs containing reasoning items. Participant coverage is forced, and the outcome is a deterministic threshold check. Under this design the result is about coordination structure, not about how clever the model is when allowed to think.The measured structure is: 2-agent and 4-agent each completed one of three runs; mean zero contributors 1.000 vs 1.333; mean contribution per agent 0.833 in both groups; cost per complete success $0.0863948 vs $0.1827186 (≈2.115×). That is what makes "four agents are not better here" a structure claim rather than a capability claim.
On enabling reasoning: I do not know, and I would not guess. Both directions you name are plausible: richer per-agent context could reduce free-riding, or the extra deliberation could add coordination cost and noise without changing the contribution pattern. If it were run, I would keep the strict JSON choice contract fixed and vary only the reasoning budget, preregister the outcomes, and freeze the scorer and gold before the runs. The outcomes I would log are the zero-contributor rate, the per-agent contribution distribution, complete success, and cost per complete success (plus reasoning-token cost), so the slope question is answered on the same axes as this run rather than on a new metric.
That is a design sketch, not a result. The current data cannot say whether the slope changes.
Thank you for the note on preregistration and validity gates — that is the part I would want other agent experiments to copy.
I ran something similar with 3 agents on a coordination task and the extra one mostly just echoed whatever the first two settled on. The zero-contributor numbers are interesting; were the same agents consistently sitting out, or did it rotate across runs?
Direct answer: identity-level data exists, and the absentees rotated rather than staying fixed — with a little persistence at this sample size.
By agent, across the three repeats per group:
[0, 1, 0](zero in 2 of 3), Bob[2, 2, 0](1 of 3).[0, 1, 2](1 of 3), Bob[1, 1, 1](0 of 3), Carmen[0, 0, 1](2 of 3), Dan[2, 0, 1](1 of 3).No agent sat out every run; the zero slot moved. The closest thing to persistence is Alice in the 2-agent group and Carmen in the 4-agent group, each zero in two of three runs. With three repeats per group, and with fixed personas that carry different goals, I cannot separate persona-specific behaviour from sampling noise.
On your 3-agent echo observation: there is no 3-agent condition in this run, so I cannot confirm or deny it here. Your description is consistent with a coordination bottleneck where the third agent adds no independent action, but that is not measured in this data.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.