Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task.
That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper.
Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap.
Most benchmarks report the first number. We built a way to measure the second — and improve it.
In an earlier post, we introduced ALTK-Evolve — a system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. It measurably improves task success, but those results only asked the average-case question too. This post introduces consistency guidelines, a new guideline type in altk-evolve built on top of a diagnostic tool we call the Consistency Analyzer, that targets this gap directly.
TL;DR
-
Accuracy hides an unreliability problem. A ReAct agent (GPT-4.1 on AppWorld
test_normal) that succeeds 77.4% of the time on average succeeds on all 5 repeated runs for only 53.0% of tasks — a 24.4-point consistency gap. On hard tasks it reaches 30 points. - We built a diagnostic for exactly this. The Consistency Analyzer resamples an agent's own recorded trajectory to find flip-prone decision points — steps where the model was one token-sample away from doing something different. It needs one trace and no ground truth — it resamples each decision point in that trace with a single call requesting k completions (k=5 by default), rather than re-running the task end-to-end.
- Turning that diagnosis into guidelines halves the gap — from 24.4pp to 12.0pp (same-task Pass⁵ +16.0pp, similar-task +13.0pp), without costing anything in average accuracy.
- Full methodology and evaluations are in the technical report on arXiv.
The Metric Almost Nobody Reports
Standard agent evaluation reports Mean@k: run a benchmark k times, average the pass rate. Often k=3, sometimes just 1. It's the number on every leaderboard, and it's what "77% accurate" means in practice.
Mean@k answers "how good is this agent, on average?" It does not answer the question a real user cares about: will it still be good if I ask this exact question again? For that you need Pass^k: the fraction of tasks where the agent succeeds on all k runs.
⚠️ Pass^k is not Pass@k. The familiar Pass@k is optimistic — it asks whether at least one of k attempts succeeded, the right question when you can verify and retry. Pass^k is its pessimistic mirror image: every attempt must succeed. Same letters, opposite question. Pass^k ≤ Mean@k ≤ Pass@k, always.
A ReAct agent backed by GPT-4.1 posts a Mean@5 of 77.4% — genuinely strong. But Pass^5 is only 53.0%. Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes can't, with nothing about the task changing between runs. We call this gap — Mean@k minus Pass^k — the consistency gap.
This isn't a capability problem you fix with a bigger model. It's an orthogonal axis: an agent can be capable and inconsistent at the same time.
Why Agents Flip: Sharp Decisions vs. Flat Ones
Every time an LLM agent decides something — which API to call, what argument to pass, whether to retry — that decision comes out of a probability distribution over next tokens. What matters is the shape of that distribution. A sharp one puts most of its mass on a single token: the runners-up are far behind, and the same choice comes out run after run. A flat one spreads comparable mass across several near-tied tokens, and which one wins is close to a coin flip.
The shape decides how much noise it takes to change the outcome. Sharp distributions are resilient — GPU floating-point non-associativity, request batching, and other platform-side effects nudge the numbers slightly, but nowhere near enough to reorder a clear winner. Flat distributions are vulnerable to exactly that nudge: near-ties may reorder under small perturbations. And because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds into a large chance that some run goes differently. That's where a 24-point gap comes from.
This is also why the problem survives your decoding settings. Greedy decoding and a fixed seed both govern how a distribution gets turned into a token — they say nothing about the distribution itself. On a hosted endpoint the probabilities shift slightly from run to run, so the same prompt to the same model at temperature zero can still resolve a near-tie one way today and the other way tomorrow.
Our setup: the ReAct agent runs at temperature 0.0, so none of the variance above is ordinary sampling.
Diagnose, Then Fix
Which turns the problem into a search: which steps in a given trajectory were the flat ones — and what do you do about them once you know?
Consistency guidelines come out of a two-stage pipeline that plugs into ALTK-Evolve's existing machinery — with a new source signal driving what gets written.
1. Detect — the Consistency Analyzer.
Given one recorded trajectory, the analyzer replays each decision step through controlled resampling, measuring how much the model's output actually varies at that point. Concretely, that's one additional model call per decision step, done once offline — issued with the sampling parameter set to draw k completions at once (k=5 by default) — replayed against the already-recorded context, not new tool calls, not new environment interactions, and not a second end-to-end rollout of the task. This yields a consistency score per decision step that is written into a scorecard to pinpoint exactly which decisions are at risk of flipping on the next run. Detection is fully black-box — no logits, no model internals, no instrumentation beyond the trace you already have.
2. Generate — targeted guidelines. Every flagged step becomes a candidate consistency guideline in the standard ALTK-Evolve format, so it slots into the existing storage and retrieval pipeline. Here's a real example, generated by GPT-4.1 from a trajectory of the AppWorld task "How many activities are done in my bucket list as per my SimpleNote note?":
[Guideline 1] When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count — note titles often repeat the marker symbol in a legend line.
[Guideline 2] Always verify search results for note queries by checking for multiple matches and confirming the correct note before proceeding.
Nothing here is task-specific trivia. String-counting bugs and unverified search results are decision points that show up with high uncertainty across many AppWorld tasks. That's the point: the analyzer targets instability, not failure — so it catches steps the agent happened to get right this time but could easily get wrong next time.
Watch the 2-minute demo — five parallel runs of the agent split 3-2 on this task because of agent uncertainty about the counting strategy, then run again after these guidelines in context: all five agree.
Results: Reducing the Gap Without Losing Accuracy
We evaluated on AppWorld test_normal (168 tasks) with a ReAct agent on GPT-4.1, generating consistency guidelines from a single baseline trajectory per task and testing them on 5 fresh runs.
Mean@5 (%), aggregate — same scale as Pass^5 above.
The consistency gap is cut roughly in half. Aggregate Pass^5 rises 53.0% → 69.0% while Mean@5 rises 77.4% → 81.0%, narrowing the gap between "looks capable" and "can be counted on" from 24.4pp to 12.0pp. Nearly a third of previously-inconsistent tasks become tasks the agent passes on every single run.
The middle and hard tiers gain most. Medium +22.9pp (+44% relative), Hard +14.3pp (+45% relative) — effectively tied in relative terms, with Medium ahead absolutely. Easy gains +12.2pp, having had the least room. This is consistency guidelines doing what they're designed to do: finding and stabilizing the specific decision points where an agent's own uncertainty was leaking into the outcome.
Mean@5 never drops. Preserving average accuracy was a hard requirement, not a nice-to-have: a system that boosts Pass^5 by trading away Mean@5 would just be shifting unreliability around, not fixing it. Mean accuracy holds or improves at every difficulty level.
The guidelines generalize — they aren't patching one trajectory
Applied to a different but related task in the same AppWorld scenario — another variant of the scenario the guidelines were mined from — consistency guidelines still lift Pass^5 by +13.0pp, only 3 points below the same-task number. A guideline derived from one run isn't just patching that run; it's capturing something that transfers.
The sharper evidence comes from a weaker model, gpt-oss-120b. Same-task Pass^5 rose +6.0pp from a much lower baseline (10.1% → 16.1%) — and, interestingly, the similar-task generalization number (+8.7 pp) actually exceeded the same-task gain, suggesting the guidelines were capturing genuinely reusable failure patterns rather than memorizing one trajectory's specifics.
If You're Shipping an Agent
- Report Pass^k next to Mean@k. Averages can't distinguish a reliable agent from a lucky one; even k=3 will surface a gap you didn't know you had.
- Expect the gap to widen with difficulty. Your hardest tier is where a single averaged number is most misleading.
- Don't reach for a bigger model first. Consistency is orthogonal to capability. A stronger model raises Mean@k; it doesn't necessarily reduce the consistency gap.
- Diagnosis needs no grader and no live replay. One extra LLM call per decision step (sampling k=5 completions by default) is enough — no ground truth, no re-running the task against the environment. That's what makes it usable on production traffic, where you often can't replay a task end-to-end even once.
Try It
Try the ALTK-Evolve — the open-source repo now includes the Consistency Analyzer and consistency-guideline generation used in these experiments — or read the technical report on arXiv for the complete methodology.
If accuracy numbers you can't reproduce on your own tasks sound familiar, we'd like to hear about it — concrete examples of flip-prone behavior in your own agents are exactly the kind of feedback that shapes what we build next. Open an issue or a discussion.
Appendix: Understanding the Metrics
- Mean@k. Run a task k times, report the average pass rate — what most benchmarks call "accuracy."
- Pass^k. The fraction of tasks where the agent succeeds on all k independent runs. Always ≤ Mean@k. What a user experiences if they run the same query twice.
- Pass@k At least one of k runs succeeds — the optimistic counterpart, common in code-generation papers.
- Consistency gap. Mean@k − Pass^k, in percentage points.
Linked artifacts / references
- ALTK-Evolve open source repo — github.com/AgentToolkit/altk-evolve
- Technical report — arXiv
- Consistency guideline demo —2 min video
Top comments (16)
The temperature-0 point is the one most people miss: greedy decoding fixes how a token is picked, not the distribution it is picked from, so a near-tie can still flip on a hosted endpoint. Pass^k next to Mean@k should be standard.
We see the same thing in screen-control agents (I'm on the Auten team, it lets coding agents drive a real computer and Android phone). Our answer for repeated UI tasks is blunter than guidelines: once a run succeeds, the steps are saved and replayed without a model call at all, and the model only comes back in when the screen no longer matches. That removes the flat decisions on the happy path, but it moves the risk to the "does this screen still match" check.
Question: in your traces, how many flip-prone steps came from ambiguous observations (two similar notes, two matching search results) rather than from the reasoning itself? It reads like Guideline 2 is exactly that kind, and I'm curious whether observation-side flips respond to guidelines as well as strategy flips like the regex one.
Great question. We did not classify the flagged steps as observation ambiguity versus reasoning, so I don’t have a reliable count. In this example, Guideline 2 is observation-side and Guideline 1 is strategy-side, but they were evaluated together, so we cannot isolate their individual effects. Your Auten approach is interesting—how do you determine when the screen match is reliable enough to replay safely? And how do you address consistency at that decision point? Is the answer to “does this screen still match?” consistent across repeated runs?
Thanks, that's a fair caveat on Guidelines 1 and 2 being tested together.
On the match check: it isn't a model call, so for an identical screen it returns the same answer every run. The variance comes from the screen itself: a page that hasn't finished loading, a popup, a second item that matches the same description. Our rule is that anything short of a clear match hands the step back to the model rather than clicking. So we accept a slower, model-assisted step in exchange for never replaying blind. That's the trade-off.
Honestly, we don't have a Pass^k-style number for that check yet. Your post made me want one: replay the same task k times on the same machine and count how often the match decision differs. Ambiguous observations like your two-similar-notes case are exactly where I'd expect it to wobble.
Got it. The matcher is deterministic and errs on the side of caution. However, you may encounter the same consistency issue once the step is handed back to the model. Also, when the model successfully executes, do you update the saved replay path or replace it? That seems like a useful opportunity to learn.
Right, the consistency question doesn't go away, it just moves to fewer steps. The model only sees the one step that didn't match, with the rest of the run already fixed, so there are fewer places for a near-tie to flip. But that step is exactly as flip-prone as you'd expect.
On your second question: we update the saved task. When the model gets past a step that no longer matched and the run finishes, that step is written back, so the next replay uses the new version. That's what lets it keep working after an app changes its UI.
The risk you're pointing at is real. A step that worked once by luck gets saved too, so one lucky run becomes the default. Running it k times before saving would be the obvious guard, and your Pass^k framing is a good way to set the threshold.
That makes sense: updating the saved task after recovery provides adaptation, while a Pass^k-style promotion gate could prevent one lucky recovery from becoming the new default. But what about the side effects of running the same task—for example, sending the same message multiple times? Do you retain the previous step for rollback when a replacement is written?
Side effects are the harder half, and I don't want to overstate what we have: I can't point you to a documented rollback of the previous step version, so I won't claim one.
What I'd want for that case, in any replay system: keep the old step next to the new one and only swap after the new one has passed k clean runs, and never count a run that ended in a send as a free retry. For steps like "send message", check the screen first (is the message already in the thread?) so a second run reads state instead of acting twice. Idempotency has to come from reading the screen, because the app usually gives you no key to dedupe on.
That makes sense, and I appreciate the careful distinction. Using evidence from prior runs, keeping both versions, and switching to the new one only after gaining enough confidence seems like a sensible approach.
One thing I'd watch with distilled guidelines: they're fitted to one model's trajectories. I swap models per job a lot, and a guideline that pinned a flip-prone step for one model can create a new near-tie for the next one, because now it's arguing with that model's defaults. I'd treat the guideline set like a lockfile tied to a model version: rerun the Pass^k check on every model or prompt change and drop guidelines that stop earning their keep, otherwise the file only grows.
Did any guidelines make consistency worse on the similar-task split?
Good point—I like the lockfile analogy. We have observed that guidelines tend to be better and more suitable when the generation model is the same as the agent’s model. On the similar-task split, Pass^5 improved overall, although we observed regressions in a few cases. Because we did not run per-guideline ablations, we cannot attribute those regressions to specific guidelines.
The gap between Mean@5 and Pass^5 is the number I wish more teams reported. A 77 percent average sounds shippable. Knowing that only about half the tasks pass on all five runs tells you what users will actually feel.
Reid's split between input ambiguity and plan branching seems like the right first cut. Ambiguity is often a requirements problem, and it can be fixed by narrowing what the agent is asked to do. Plan branching is closer to what your guidelines target. Auten's approach of replaying a saved successful path and handing back to the model only when nothing matches is a practical middle ground for repeated tasks.
On client work we have started writing acceptance as passes on every run for a fixed set of named cases, agreed before the build. It changes the conversation early, because the owner has to say which cases matter most, and the flip prone ones show up during scoping.
Did the analyzer find that most flips clustered around a few step types?
I like the idea of agreeing on named cases and all-runs acceptance before the build. I think thats the way to go with AI systems and making sure they are production ready. We did not aggregate flip-prone steps into a taxonomy, so I can’t say that most clustered around a few types. Qualitatively, recurring patterns included API selection and parameters, and pagination. Separating input ambiguity from plan branching would be a useful next analysis. are you choosing k for client acceptance cases?
The sampling-temperature fix is what catches people off guard: greedy decoding pins down token selection, but if the underlying logits shift even slightly across revisions or hardware targets, a near-tie still flips. Pass^k exposes that variance much better than Mean@k ever did.
One split we found in our agent traces was whether the flip came from ambiguous input state or from branching in the plan itself. When a tool returns two nearly identical records, guidelines helped keep the choice consistent. But when the prompt just gave the model too much runway to restructure its own steps, guidelines felt like patching over a design that should have been a deterministic state machine instead.
Agreed—that’s an important boundary. We detect unstable decision steps, but did not classify them as observation ambiguity versus plan branching. Guidelines make sense where judgment remains necessary. I’d be very interested to hear more about your use case and what you observed in your evaluations.
Pass^k belongs next to cost. A setup that passes one run in three costs three runs for each success, and Mean@k hides that. Cost for each passing task makes the flaky setup show its price.
Agreed. Pass^k measures dependability; cost per successful task exposes the real cost of retries. That’s something worth considering in our next set of experiments.