The 480 Outputs That Never Became Actions

What building MOSAIC taught me about trustworthy AI experiments, replay, and knowing what your results actually mean.

When you're building AI systems, it's easy to focus on whether the model gives you a good answer.

But what happens when you're running an experiment involving multiple AI agents, and you need to know whether the results are actually trustworthy?

That's the question I've been exploring while building MOSAIC, a research project focused on studying the behaviour of populations of language-model agents.

The project has taken me through event-sourced architectures, model-generation caching, replay systems, strict validation, experimental design, and quite a few unexpected problems.

One recent experiment taught me something particularly important: a system can execute exactly as designed and still fail to produce the observations needed to answer the research question.

Here's what happened, and what I learned from it.

First, why am I building MOSAIC?

Imagine you have a society of AI agents working together in a shared environment. You replace one agent's underlying model with a different model, and suddenly the society behaves differently.

What caused that change?

Was it the model substitution? Was it ordinary variation between generations? Or would the same change have happened even if you hadn't modified anything?

Simply running the experiment again doesn't necessarily answer these questions. Model outputs can vary, and the state of the agents, their communication history, and the surrounding environment can all affect what happens next.

MOSAIC is my attempt to build the research infrastructure needed to investigate these questions more rigorously.

The idea is to record what happens, preserve the conditions of an experiment, validate the execution, and create a defensible basis for comparing outcomes.

Before making claims about multi-agent behaviour, I want to be confident that the system measuring that behaviour is doing its job.

Building the infrastructure before interpreting the behaviour

One of the central design choices in MOSAIC is its event-driven architecture.

Instead of treating a run as a collection of loosely connected outputs, the system records events in an append-only log. Its state can then be reconstructed by processing those events in order through a deterministic reducer.

That gives us a recorded history of what happened during a run and a way to check whether subsequent state reconstruction is consistent with that history.

The runtime also records important information about model identity, generation parameters, protocol settings, and other execution details. Model identity is tied to a digest rather than relying only on a model tag, which might not always identify the same underlying model artifact.

These details can sound like implementation minutiae, but they become important when you're trying to explain a difference between two experiments.

If you can't establish which model ran, what inputs it received, or how the system reached its recorded state, interpreting the result becomes much harder.

Replay is useful, but it doesn't prove everything

Another important component of MOSAIC is its record-and-replay system.

During recording, model generations are saved alongside the information needed to identify and validate the corresponding requests. During replay, the system can use those recorded generations instead of making fresh model calls, then check whether the recorded execution can be reconstructed consistently.

This is useful for debugging, validating the execution pipeline, and checking that the experimental driver follows the same recorded trajectory.

But there's a distinction worth understanding:

Replaying an LLM execution is not the same as independently reproducing the model's behaviour.

When a replay system uses a saved generation, it isn't asking the model to generate that output again. It is using the recorded output.

So a passing replay tells us that the reconstruction matches the recording under the checks performed. It doesn't establish that a fresh model call would produce the same response.

I think this distinction matters far beyond MOSAIC. Whenever we're evaluating an AI system, we need to be clear about what our validation process actually proves.

Then came an experiment that didn't produce usable observations

The recent RQ-01 null pilot gave me a concrete example of why all this matters.

The pilot was designed around two separately accounted-for populations using different local language models.

The first population, S1, used llama3.1:8b. It completed 22 source units and produced 66 usable pairs. All 66 pairs were processed by the project's Tier-0 measurement utility.

The second population, S2, used qwen2.5:7b.

Its retry stopped after three completed source units. Under the registered stopping rule, the experiment had to stop after three consecutive source units in which both planned branches failed to realize usable actions.

That is exactly what happened. All six S2 branch pairs were censored for non_realized_arm, leaving zero usable null observations for S2.

The interesting part is that the stopping mechanism worked as intended. The system did not need someone to intervene manually once the registered condition was met.

But successful execution of a stopping rule isn't the same as successfully answering the research question. S2's restricted campaign reached its registered stopping condition, while the overall two-population pilot remained incomplete.

So what was preventing those observations from becoming usable?

The 480 outputs that never became actions

The next step was to investigate the actual generation records.

Across the nine S2 retry run directories, there were 576 generation records. After accounting for replayed branch-prefix duplicates, the investigation identified 480 distinct generations.

All 480 distinct generations exhibited the same relevant output pattern: a bare write_artifact prefix followed by a JSON object.

The output contract was explicit. Each response had to contain exactly one JSON object, with no additional text before or after it.

Because of the prefix, the strict parser rejected the responses as not_json.

This is a useful distinction. The investigation found that the JSON object itself could be parsed when the prefix was removed from the recorded text. But that doesn't mean the original responses complied with the protocol, and it doesn't tell us what would have happened if the responses had been accepted.

The registered execution pipeline never reached the point where the proposed actions could be evaluated by the arbiter. We therefore have no evidence that removing the prefix would have led to accepted actions or successful agent behaviour.

The immediate failure category is clear. The underlying reason for the repeated prefix is not.

What the investigation ruled out—and what it didn't

I also investigated the recorded request configuration for the S2 retry and compared it with a diagnostic sample from S1.

The shared configuration settings matched across the compared records, including the contract and policy blocks, message structure, sampling configuration, protocol identity, provider version, and recorded server configuration.

However, the model identities differed, and the per-generation seeds differed as well. The seed sets were distinct between the two samples.

The server-side chat template and the fully rendered prompt were not preserved in the existing records. As a result, the investigation could not establish whether the template, model behaviour, or another unrecorded aspect of the generation process explained the prefix.

This is why I don't want to jump to the conclusion that one model is incapable of coordinating, or that a particular template caused the issue. The evidence doesn't establish either claim.

The right conclusion is narrower: the S2 retry repeatedly violated the required output format under its recorded conditions, and the mechanism behind that behaviour remains unresolved.

What this taught me about experimental validation

There are three lessons I'm taking away from this experience.

First, a failed experiment isn't necessarily a failed instrument.

The S2 run didn't produce usable null observations, but it exposed a concrete failure mode. The validation system detected the outcome, classified the unusable pairs, and stopped when the registered condition was met.

That gives us useful information about the instrument, even though it doesn't answer the original scientific question.

Second, we need to separate engineering evidence from scientific evidence.

Passing tests, validating a server, replaying a recorded run, and measuring agent behaviour are different activities.

Each tells us something different. A software test can establish that a component behaves as expected under the tested conditions. A replay can establish that a recorded execution is reconstructed consistently. A behavioural experiment needs usable observations and a measurement procedure appropriate to its research question.

Treating these as interchangeable would make it much too easy to claim more than the evidence supports.

Third, we shouldn't change the rules just because an experiment produced an inconvenient result.

MOSAIC has explicit research decisions governing the current pilot. Those decisions require the populations to remain separate and prohibit rerunning the censored S2 units or replacing the registered Qwen model under the existing authorisation.

That means I can't simply remove the prefix, change the parser, and treat the resulting execution as though nothing about the experiment changed.

A modified setup might be worth investigating in the future, but it would need its own explicit research decision and a clear account of what remains comparable with the existing results.

The point isn't to avoid fixing problems. It's to avoid quietly changing the experimental conditions after seeing the results.

Where MOSAIC stands today

MOSAIC has grown into a substantial experimental instrument, with event-sourced state, model-generation recording, guarded replay, validation, society execution, and branching capabilities.

The recent pilot has also produced real evidence about the instrument's behaviour under failure conditions.

But there are important limits to what I can claim today.

S1 has 66 usable pairs processed by the Tier-0 measurement utility. S2 has zero usable null observations. The primary metric-selection procedure has not yet been implemented, and the registered two-population question remains unresolved.

The current results do not establish the S2 divergence noise floor, demonstrate a causal effect of model substitution, or show that one model is better at multi-agent coordination than another.

That might sound like a modest conclusion after so much engineering work. I actually think being able to state these limitations clearly is part of building a trustworthy research system.

What comes next?

The next step isn't to announce a scientific result that the data don't support.

It's to decide, through the project's research-governance process, how to proceed with the current study and whether a separately authorised exploratory analysis of the existing S1 measurements would be useful.

Beyond that, MOSAIC needs further work on measurement validity, statistical treatment, and carefully controlled experiments that can establish what causes observed differences in agent behaviour.

Those are still open problems.

For me, the central lesson from this project so far is simple:

A trustworthy AI research system should help us understand not only what happened, but also what we're justified in concluding from it.

Sometimes an experiment gives you the result you were looking for. Sometimes it exposes a limitation you didn't know you had. And sometimes it stops with no usable observations.

In each case, the responsibility is the same: preserve the evidence, explain what it supports, be honest about what remains unknown, and make the next decision deliberately.

That's the standard I'm trying to build into MOSAIC.

And the research is still ongoing.