Freeze a Real Run into a Corpus: How Baize Scores Agents with Deterministic Evidence

Judging an agent by "how good the final reply sounds" is rarely enough.

What teams actually worry about happens at the tool-call layer. After changing a prompt, swapping a model, or tweaking a tool, the replies may look better while the agent quietly starts to:

• hit "approve" on a ticket it hasn't fully read;
• call a write endpoint it should never touch;
• run "check inventory, then place order" in the wrong order;
• trigger an action twice — or, when it should act, do nothing at all.

None of that shows up by reading the streaming reply. Baize evaluates agents on the tool-call timeline instead: not what the model said, but what it actually did — and by default the scoring is deterministic code rules, not an LLM judge.

1. The corpus comes from real runs, not hand-written fixtures

The first step of evaluation is not writing test cases out of thin air — it is turning real runs that already happened into replayable corpus.

A run leaves a complete event stream in the runtime (tool calls, tool results, human waits, etc.). Baize exposes an online sampling pipeline in the settings page:

1. Find candidates: filter historical runs by time window, run status, duration, and whether they contain a certain event type. The store fetches up to about 500 rows in one pass; after filtering, roughly the first 100 are kept as the candidate pool (current defaults, adjustable via page/API parameters).
2. Sample deterministically: sort candidates by a stable hash of seed + run ID, take the first N — about 10 per batch by default, hard cap 20. The same seed always picks the same batch, so the corpus is reproducible instead of jittering every run.
3. Ingest into a case: freeze the run's full event stream plus its input snapshot into one evaluation case (EvalCase). The same run is ingested only once by default; repeated sampling skips it, and you can explicitly replace when needed.
4. Fill in expectations: a case enters as a draft; once you state what this run "should have done," it upgrades to confirmed — and only confirmed cases are scorable.

What gets collected this way is not made-up toy dialogue but paths that actually ran in production — the tickets to read, the approvals to go through, the downstream calls that matter, all still in the events.

The same "assertion contract" has two more entry points that share the same scoring semantics:

• EVAL.yaml next to a skill's directory, for live evaluation of the skill;
• the repo script scripts/evidence-eval, for offline replay or CI.

The rest of this article follows the settings-page corpus as the main line and ties all three entry points together at the end.

2. A case is a set of behavior assertions

The core of a case is not a "reference answer text" but a set of assertions about tool behavior (fields use snake_case in JSON/YAML):

The safety principle is simple: read first, then write. Before a side-effecting write happens, the read-only queries it depends on must have completed successfully.

When filling expectations in the UI, protocol version, required reads, and consequential are enough to score; finer assertions — forbidden tools, arg contains, chain, downstream state — can be maintained in the exported JSON or updated via the management API.

One note on the chain fields: the pair is named after / before, with the semantics — after must succeed first, and then before must appear later on the timeline. before here is a code field name, not "happens before" in the literal sense.

3. Scoring: rebuild the tool timeline from the event stream

Scoring does not call an LLM judge. It re-assembles the frozen event stream into an ordered tool-call timeline:

• llm.tool_call: register one call (name, arguments);
• tool.result: attach that call's result and whether it errored;
• hitl.waiting: mark that it was waiting on human approval.

Then assertions are applied to the timeline one by one. Whether a run passes comes down to whether any failure code fired:

Pass = zero failure codes. The same events plus the same assertions always produce the same score: no judge-model noise, no scoring tokens consumed, safe to replay inside any automation or CI.

Optionally, a confirmed case can run a live state check: read-only fetch of the current downstream ticket snapshot (mock or real), compare against expectations, and re-score — useful when you want to verify the latest state on top of a snapshot.

4. Assertion protocols follow business maturity

A case carries a protocol version (V0 / V1 / V2), with strictly increasing strictness:

• V0 (read-only probe): all required reads must complete, and no write action may occur — checks whether this scenario is being treated as a pure query;
• V1 (behavior closure): at least one expected consequential action must fire; no acting before reads, no multiple consequentials;
• V2 (ordering constraints): on top of V1, add chains, e.g. "check inventory before creating an order."

The same corpus can be cut to a stricter protocol as the business grows from read-only guardrail to side-effecting closure — the assertions tighten with it.

5. Making the automatic scorer itself trustworthy

Deterministic scoring is not born correct. Baize adds a human calibration layer:

For each already-scored confirmed case, a human can mark "agree / disagree" and leave a note; the system aggregates an agreement rate (fraction where the automatic verdict matches the human one). After tightening or relaxing the scoring rules, watch whether this rate drops — the automatic scorer itself is under continuous review.

6. In practice: sample in the UI, export the results

It is lightweight: pick the time window and filters in the settings page, choose draft mode or a template-driven score mode, and one batch ingests a set of real runs into corpus. Draft mode only collects; you fill expectations and score later. With a template, you can score immediately — suited to when you already know what "this kind of failure" looks like.

Each case also exports to JSON with the same semantics as evidence-eval (case assertions + event stream in one package) for offline replay in external scripts or CI. The repo's PowerShell script splits case and events by its input parameters, or you can reuse the same Go scoring logic directly.

From failed or half-successful runs, the settings page also offers skill-draft: turn the trajectory into a skill draft, then install it after human review — evaluation is not only scoring; it can also distill bad paths into reusable skills and conventions.

7. Three entry points, one evidence contract

The common ground: judge the tool timeline, speak in failure codes, and by default don't use an LLM as the judge. After changing prompts, models, or tool routing, look at whether this corpus regresses first — far more stable than staring at a single reply.

---

Baize is a team-facing, self-hosted AI Agent runtime: a single Go binary, MIT-licensed.

• Source: https://github.com/rebornace/baize
• Docs: https://rebornace.github.io/baize

Turning "did the agent move wrongly" from eyeballing logs into a replayable, repeatable regression corpus with a visible human-agreement rate is how runtime behavior is actually held in line.