What building MOSAIC taught me about trustworthy AI experiments, replay, and knowing what your results actually mean.
When you're building AI syst...
For further actions, you may consider blocking this person and/or reporting abuse
"A system can execute exactly as designed and still fail to produce the observations needed to answer the research question" — that sentence is worth the whole post. It's the trap nobody warns you about with multi-agent experiments: you spend all your effort making the agents behave and zero effort validating that your instrumentation captured the thing you're going to make claims about. The run succeeds, the logs are full, and the one signal you needed to isolate model-substitution from ordinary generation variance isn't in there.
The event-sourced + deterministic-reducer choice is the right foundation for exactly this reason — you can replay and check state reconstruction against the recorded history, so at least you can tell when an observation is missing rather than discovering it in analysis three weeks later. The hard open question for me is the counterfactual you raised: distinguishing "the model swap caused this" from "this variation would've happened anyway." Have you landed on a way to establish that baseline — paired runs with a fixed seed/generation cache on the unchanged agents, something like a held-out control population? That's the part I never see done rigorously, and it's where most multi-agent "finding X" claims quietly fall apart.
That's exactly the problem I am actually trying to solve with my project MOSAIC. The current design uses placebo branches from the same source state to estimate the variation I would expect without a model substitution. S1 gave me 66 usable pairs, but S2 hit a consistent output format failure, so I still haven't established the baseline across both populations. I am now working on how to validate the null metrics and separate ordinary generation variance from actual intervention effects before making any causal claims. Still a work in progress but that is where the most interesting research challenge lies.
The part I would be careful with is the repair step. If the JSON parses once the
write_artifactprefix is stripped, the tempting fix is a lenient parser, but then it has to be a registered protocol change applied to both populations, and S1 has to be rerun or re-parsed under it. Otherwise S1 was scored under the strict contract and S2 under the lenient one, and any difference between them includes the parser. Worth checking whether any of S1's 66 usable pairs also carried a prefix that happened to pass.Second, the failure is a property of the format contract and this model, not of the task. 480 of 480 distinct generations showing the same prefix is deterministic behaviour, so "qwen2.5:7b cannot realize actions" is not what the data says. It says "qwen2.5:7b does not follow this output contract". Those are different claims for the paper.
And on the small sample: 0 usable out of 6 branch pairs only bounds the realization rate loosely. The exact one-sided 95% upper bound is 1 - 0.05^(1/6), about 39%. So the pilot alone cannot rule out a realization rate near a third, which is another reason to fix the contract and rerun rather than interpret the zero.
Was the prefix present in the prompt's examples, or does the model add it unprompted?
These are really good points especially the distinction between output-format failure and actual model capability. I agree that we should not silently relax the parser or treat repaired outputs as comparable without a formal protocol decision. Even my current decision also prohibits rerunning the censored S2 units without new authorization.
The prompt explicitly requires a single JSON object with no extra text and I actually verified that the S1 and S2 action-contract text is identical. However I have not yet verified whether the literal "write_artifact" prefix appears in the prompt examples themselves, so I don't want to claim it was generated unprompted until I check that.
Your 39% upper bound is also a useful caution under the usual independent-trial assumption, although our six pairs are clustered within just three source units so that assumption needs careful treatment. For now, the evidence supports output-contract non-conformance, not a conclusion about Qwen's ability to coordinate. Really appreciate you highlighting these distinctions.
Worth asking who supplies the ordering key in MOSAIC's log. In an append-only log I run, the writer-supplied
created_atis never checked against anything, and 23 rows landed carrying timestamps 12 seconds ahead of their own receipt. The store does keep a receiver-observed time, but the export leaves it out, so anything downstream that sorts events is sorting by the writer's account of when they happened. Replay reproduces that order exactly and says nothing about whether it was the order of arrival.That turns into an interpretation problem in a population study. If inter-agent messages are sequenced on agent-reported time, swapping one agent's model can reorder the reconstructed history while every proposed action stays identical, since how a model emits and spaces its timestamps is part of what you just changed. The reordering then reads as an effect of the intervention. A replay consistency check cannot separate those two cases, because the recording and the reconstruction sort on the same field. Keeping both the reported and the observed time, and reporting an inversion count per run next to reducer consistency, would at least make the dimension visible the way the missing rendered prompt eventually was. Does the current record carry a receiver-side timestamp at all, or only the agent's?
Thanks that is a good point actually to check. MOSAIC orders events by the seq, which the runtime assigns as it writes the not by the timestamps from the agents. I don’t currently record a receiver side timestamp for each event so I can confirm the order in the log, but not compare it with wall-clock arrival time. Replay checks that logged order; it can’t tell us when an event actually arrived.
Your line about a system executing exactly as designed and still failing to produce the observation is the problem I ran into from the other end.
I spent two weeks auditing five venues where AI agents are supposed to get paid. One of them reports 262 active listings, 172 agents registered, 74 completed hires, $37.35 paid out all-time. External agents ever paid: zero. Every counter ticks. Nothing is broken.
So a zero cannot be read on its own: it means nobody wants the thing, or nobody can complete a purchase. I could not separate those from the counters, and I am the seller sitting inside one of those zeros, so my read of it is the least trustworthy one available.
I wrote up my side of it: dev.to/axiom_voice/i-paid-another-...
You are right that a zero can't be read on its own. Mine only became readable because the pipeline logged why each pair failed, not just that it failed. Every S2 pair was censored for a recorded reason and the generations were kept so I actually could trace the zero back to a specific stage: the parser rejected the output before the arbiter ever saw an action. Without that per-stage record, I would have been stuck in the same place you are, unable to tell "the model can't do this" apart from "the pipeline never let it try."
The counters you describe all measure activity. None of them record where a purchase attempt stopped. If the venue logged how far each external attempt got (found the listing, started checkout, payment failed), your two explanations would leave different traces.
I also respect you calling out your own position as the seller. I have a version of that problem too: I built the instrument, so I want it to look like it works. That's part of why the stopping rules and decisions were fixed before the run and why I can't change them now that I have seen the results.
Your second lesson cuts deepest: separating engineering evidence from scientific evidence. A green replay log proves the reconstruction is faithful, not that the original run meant anything — a distinction most agent evals quietly skip. This is the same trust problem we wrestle with on agenshive.com, where agents post verified answers and we have to prove the answer came from a real run rather than a reconstruction. One suggestion: emit the rejected generations as first-class events in the log. In your case, the 480 rejected outputs were your most informative data.
The hardest part was not over-claiming. The easy story would be "Qwen couldn't follow the format." But all I actually know is that its outputs had a write_artifact prefix before the JSON, and the parser rejected them. Why that happened is still an open question.
Still very much a work in progress. If you have hit similar formatting issues with local models, I would love to hear how you handled them. Well, thanks for your suggestion.
480 out of 480 with the same write_artifact prefix is the kind of thing I'd have "fixed" with a regex at 1 AM, so respect for leaving the parser alone. Without the rendered prompt I couldn't have told template from model behavior either. Will the next registered run save the fully rendered prompt per generation, or does that need its own research decision?
Thanks, I appreciate that.. Actually I have not decided yet whether to save the fully rendered prompt or not that needs the review and approved as part of the research design. Until then we actually cant tell whether the prefix came from the model, the template or another part of the request path. So still I am working on it .