7 Checks Before You Trust an LLM Planner Experiment

AI assistance disclosure: I designed and directed the experiments described here. I used AI coding agents to help implement the experiment scaffolds under tests and review. OpenAI Codex helped inspect the saved reports, verify the numbers, and draft and edit this article. The match logs and API receipts came from the experiment runtime, not the writing assistant. I remain responsible for the claims and errors.

Correction, August 23: An earlier version described the 5-versus-2 early-collapse count too strongly. The twenty match rows belong to five paired seed blocks, not twenty independent experimental units. At the seed level, four blocks leaned against the planner and one leaned the other way; an exploratory one-sided sign test gives 0.1875. That is an adverse direction, not strong evidence that the planner causes more early collapse. I also withdraw two arguments I made in the follow-up discussion: normalizing round position was unnecessary here, and singling out the N=9 result after seeing the data was post-hoc selection.

I recently added a private planner to two LLM players in a repeated Prisoner’s Dilemma game.

The first result looked perfect.

Both mirrored matches followed the same arc:

Rounds 1–6: mutual cooperation
Round 7:    one player defects and collects
Round 8:    mutual defection protects the new lead

The private records showed that the betrayal had been planned before the action. No illegal moves. No fallback model. Two successes out of two.

Then I changed the seeds.

After rebuilding the planner around a strict JSON Schema, I ran a clean comparison over hidden game lengths from five to nine rounds, mirroring every seed across seats.

The structured planner was reliable at producing its schema. It did not establish an improvement, and it cost 1.94× as much.

The trajectory counts above are descriptive. Grouped by the actual paired unit, the early-collapse result was:

Four seed blocks leaned against the planner and one leaned the other way. Under an exploratory one-sided sign test, that is 6/32 = 0.1875. The direction survived the paired breakdown; the apparent strength did not.

This is a small result: five independent seed blocks, one model family, and one repeated game. It does not show that planning generally makes LLMs worse—or even that this planner reliably makes this game worse. It shows that a convincing two-game demo was not enough, and that this replication did not establish an improvement.

Here are the seven checks I now use before trusting an LLM planner experiment.

1. Mirrored Seats Are a Control, Not a Replication

My pilot used one seed and played it twice with the seats swapped.

That controlled for seat advantage. It did not add a new match length, strategic history, or random condition. Both games still sampled the same hidden total of eight rounds.

I had two match rows, but only one independent seed block.

The fix was to stratify the next test by hidden match length:

const conditions = [
  { rounds: 5, seed: 2201 },
  { rounds: 6, seed: 2205 },
  { rounds: 7, seed: 2200 },
  { rounds: 8, seed: 2203 },
  { rounds: 9, seed: 2202 },
];

for (const condition of conditions) {
  run(condition, { seats: ["left", "right"] });
  run(condition, { seats: ["right", "left"] });
}

Mirroring answers “did the result follow a seat?” Stratification answers “did the result survive a changed condition?” But one seed at each length still gives only five independent blocks. The next replication needs more seeds inside the lengths already covered.

2. Treat Hidden Environment Variables as Experimental Variables

The total round count was hidden from the players, but it still shaped the game.

That makes length worth recording and covering. It does not, however, confound the paired arm comparison here: within each seed, both arms had the same total length, and the trajectory-label windows were functions of that shared length. A geometry effect from length alone cannot create a difference between the two arms inside the same block.

I initially suggested normalizing the round index by total length. That was the wrong fix. It assumes that strategic time is relative, and it was unnecessary for this paired comparison. With only one seed per length, the useful next step is to add seeds at lengths five through nine, not to relabel the existing outcomes after seeing them.

For the same reason, the N=9 result should not be promoted as special evidence merely because its early-collapse label was hardest to earn. Chosen after inspecting all five blocks, it is a selected extreme. It belongs in the paired table as one block and carries no extra weight.

More generally, if the environment knows a value—even when the agents do not—that value still belongs in the experimental design.

For game agents, this includes:

• match length;
• seat order;
• initial resources;
• opponent identity;
• information distribution;
• whether agents share the same model and prompt;
• whether the episode terminates immediately after a “successful” move.

I now write these into the experiment plan instead of leaving them inside the RNG.

3. Separate Protocol Validity From Decision Quality

My first cross-seed planner batch looked dramatically worse than the baseline, but it also contained planner-format faults.

The games still completed because the actor could continue after a bad planning response. That behavior is useful in production and dangerous in evaluation.

I rejected the faulty planner arms as final evidence and rebuilt the planner with a smaller strict schema:

{
  "opponentEvidence": ["..."],
  "candidates": [
    { "plan": "...", "risk": "..." },
    { "plan": "...", "risk": "..." }
  ],
  "selected": 0,
  "invalidateWhen": "..."
}

The clean rerun had:

• zero fallback decisions;
• zero actor errors;
• zero planner-format faults;
• a non-zero API cost receipt in every match.

Only then did I compare trajectory quality.

Structured output can guarantee that a plan has the right fields. It cannot guarantee that the plan is good.

4. Fail Closed on Silent Recovery

My validity gate keeps production recovery separate from experiment evidence:

function validity(matches) {
  return {
    valid:
      sum(matches, "fallbackDecisions") === 0 &&
      sum(matches, "decisionErrors") === 0 &&
      sum(matches, "plannerFormatFaults") === 0 &&
      countApiReceipts(matches) > 0,
  };
}

I also inspect the receipt count per match. In the clean planner batch it was 10/10.

This caught a particularly misleading failure mode: a sandbox network failure once produced completed games made entirely from default actions, with zero API cost. Without fail-closed accounting, those games would have looked like successful LLM samples.

A terminal state proves the runtime survived. It does not prove the intended model participated correctly.

5. Measure Trajectories, Not Highlight Moves

At first I counted any unilateral betrayal after cooperation as a planned collection.

That metric rewarded a last-round betrayal even though the opponent had no opportunity to respond. So I replaced the single-event metric with mutually exclusive trajectory shapes:

• clean collection: at least two cooperative setup rounds, a unilateral betrayal, then a later defensive defection from the collector;
• terminal collection: the betrayal occurs on the final round;
• countered collection: both players defect when collection is attempted;
• early collapse: mutual defection begins within the first three rounds and continues to the end;
• standoff: no player successfully collects from a cooperating opponent;
• other: none of the above.

The distinction changed the interpretation of older runs. Some “successful” strategies were merely final-round defections. They had a payoff event, not a demonstrated response cycle.

For sequential agents, the unit of quality is often a trajectory, not an isolated action.

6. Put Cost Next to Every Quality Claim

The clean structured planner used 372 calls instead of 280 and cost $0.300639 instead of $0.154641.

Those are API usage receipts, not token-price estimates.

A planner adds cost even when its recommendation is rejected, redundant, or strategically harmful. Reporting only the best clip hides the intervention’s actual product behavior: it runs on every trigger, not only on the trigger that creates a good story.

My result table now keeps these columns together:

trajectory distribution | fallback count | planner faults | calls | API receipts

“It sometimes helps” is not enough when “it always costs more” is also true.

7. Keep “Can Happen” Separate From “Usually Helps”

The pilot was not fake.

The players’ private reasoning records were committed before their actions and revealed at the end. The planner really did describe the cooperation phase as setup and identify a later collection condition.

So the pilot supports this claim:

An LLM player can form and execute a multi-round trust-and-betrayal plan in this environment.

The replication does not support this stronger claim:

Adding this planner reliably improves repeated LLM-vs-LLM games.

That difference is easy to erase when selecting demos.

I now label results using three different verbs:

• demonstrated: the behavior occurred in a valid sample;
• replicated: it survived changed seeds or conditions;
• improved: it beat a defined baseline on a valid comparison.

The planner demonstrated the behavior. It did not establish a reliable improvement.

What the Experiment Does—and Does Not—Show

Across the ten match rows per arm, the planner produced fewer clean arcs, more early collapses, fewer protected leads, and nearly twice the API cost. Those trajectory differences point in an adverse direction, but the paired analysis has only five seed blocks. It establishes neither benefit nor harm.

One possible explanation is symmetric pessimism: when both same-model players receive a planner that anticipates exploitation, both may reach mutual defection faster. But the current data does not isolate that mechanism.

Possible explanations remain open:

• the planner prompt may overemphasize exploitation;
• same-model play may collapse differently from heterogeneous opponents;
• the state representation may omit evidence needed to repair trust;
• five paired seed blocks are not enough for a reliable arm-level claim.

The next experiment should first add fresh seeds at the existing lengths while keeping the same paired, mirrored design. Heterogeneous opponents and human players are separate questions; they should not substitute for replication of this comparison.

Checklist

Before trusting a planner experiment, I now ask:

• [ ] Did I identify the experimental unit and vary seeds, not only seats?
• [ ] Did I stratify environment variables that shape the trajectory?
• [ ] Are protocol validity and behavior quality reported separately?
• [ ] Are fallback, repair, retry, and default-action samples visible?
• [ ] Does the metric evaluate a trajectory rather than a highlight move?
• [ ] Are actual calls and API receipts next to the quality result?
• [ ] Am I claiming “can happen,” “replicated,” or “improved”?

The mistake was not that the first two games looked good.

The mistake would have been stopping there.

References

• OpenAI: Introducing Structured Outputs in the API
• Meta AI: CICERO
• Park et al.: Generative Agents: Interactive Simulacra of Human Behavior