On July 24, ARC Prize verified Claude Opus 5 at 30.16% on the ARC-AGI-3 public set. On August 21, NVIDIA reported the same model at 100.00 on the same set. The weights did not change. The code around them did.
In between, MIT did the same thing (August 5), a group led by Impossible Research got to 98.98 (July 15), and OpenAI tripled GPT-5.6 Sol's score by flipping two API settings (July 29). Then Microsoft published a framework that trains the model through the harness (August 18), and Google published one that gives the environment a harness of its own (August 20).
In July I wrote that self-editing harnesses have a provenance problem. This month the problem moved up a level: the benchmark score itself has no provenance.
TL;DR: On ARC-AGI-3's public set, the spread between "model in the official harness" and "model in the best harness" is 25 to 70 points, on a benchmark designed to resist exactly this. None of the 100s are verified on the private set, and every author says so. Microsoft's Agent Lightning v1.0 runs RL with the deploy-time harness owning the loop, so the harness is becoming part of the weights, and its reward-hacking section is the checklist my July post warned about. A benchmark number without a harness version, memory state and action budget attached is a self-reported claim with an unmarked type. Unmarked means self-reported.
Five harnesses, one public set
ARC-AGI-3 scores agents with RHAE (Relative Human Action Efficiency). Per level, score = (human_baseline_actions / ai_actions)^2, with the ratio capped at 1.15x the human baseline. Game scores are level-weighted averages, you must finish the last level to get full credit, and the overall number is the mean over games. A 100.00 means the agent finished every level at least as efficiently as a first-time human.
| Harness | Who | Date | Model | Public RHAE | Actions | Verified by ARC Prize |
|---|---|---|---|---|---|---|
| Official ARC Prize harness | ARC Prize | Jul 24 | Claude Opus 5 (high) | 30.16% | n/a | yes |
| Official harness, default settings | OpenAI | Jul 29 | GPT-5.6 Sol (max) | 13.3% | n/a | no |
| Official harness + retained reasoning + compaction | OpenAI | Jul 29 | GPT-5.6 Sol (max) | 38.3% | 6x fewer output tokens | no |
| Schema | Impossible Research (+ UC Berkeley, CMU) | Jul 15 | Opus 4.8 / Fable 5 | 98.98 | n/a | no |
| VISTA | MIT (Han, Hu, Qiu, Wu, He) | Aug 5 | Claude Opus 5 | 100.00 | 7,542 (humans: 17,135) | no |
| AVO | NVIDIA | Aug 21 | Claude Opus 5 | 100.00 | 6,624 | no |
The number that matters is not in the table. It is the gap between the first row and the last: 70 points, same model, same 25 games, same metric.
The official harness is not a neutral baseline. OpenAI's write-up quotes ARC's intent: an "intentionally generic harness, without tools or special features" built to make "model shortcomings more visible." In practice it discarded all private reasoning after each game action and used a rolling truncation window, so older actions vanished as history grew. Retaining reasoning and enabling compaction took Sol from 13.3% to 38.3% and cut output tokens by 6x. The harness was wiping the model's mind between moves.
So the leaderboard measures "model plus a harness built to expose the model." The 100s measure "model plus a harness built to cover for the model." Neither measures the model, and nobody has isolated which part of the 70 points is which.
The authors are unusually honest about this. NVIDIA: the AVO-versus-VISTA comparison "should not be interpreted as a controlled ablation," and the results "should not be interpreted as a direct measurement of the performance contribution of AVO." VISTA: the models "were released after the public ARC-AGI-3 games," overlap cannot be excluded, and "the private set remains the real test of generalization." Schema: "no frozen-harness or held-out-performance claim." Every 100 on that table is a public-set number on games the models may have seen in training.
What the 70 points are made of
Read the harness papers side by side and the same three components appear under different names.
Memory. VISTA keeps a "lossless visual memory" of every past observation. AVO carries forward "prior implementations, evaluation results, compiler and profiler outputs, and accumulated reasoning." OpenAI's two settings are memory settings: keep the reasoning, compact instead of truncate.
Supervision. AVO runs a monitor that watches "the broader trajectory for stagnation or repeated unproductive cycles and can redirect the main agent." That is the layer that turns a model that gives up into an agent that does not.
An action budget. RHAE squares the efficiency ratio, so wasted moves are punished quadratically. AVO's headline against VISTA is 12% fewer actions. That is a harness optimization target, not a model property.
In July I split harness work into two piles: compensatory layers that patch what the model cannot do yet, and protective layers that constrain what it must not do. I predicted pile one depreciates with every model release. All three components above are pile one, and on a benchmark built to resist static tricks they are currently worth 25 to 70 points with the newest frontier models. Either my prediction is early or it is wrong about magnitude. I will take the second reading until the private-set numbers say otherwise.
One more thing about compaction, since it is the setting that tripled OpenAI's score. In my preregistered compaction experiment, the same operation produced 3.47% false proceeds on irreversible-action gates: the agent went through a gate it should have stopped at, because the compacted context no longer carried the provenance the gate depended on. Not a contradiction. ARC-AGI-3 scores task completion; my gates scored whether the agent still knew why it was allowed to act. Compaction improves the first, degrades the second, and a benchmark only sees the first.
Then Microsoft put the harness inside the training loop
Agent Lightning v1.0 (arXiv, August 18) names something the July thread never got to: RL where the harness is not a bystander. In their words, "the harness owns this loop, while the training engine observes only a sequence of LLM request-response pairs." The deploy-time scaffold (mini-SWE-agent in their coding runs) executes the task inside Kubernetes; the trainer sits behind a gateway that looks like a normal LLM endpoint and collects the traffic.
The result is real: Qwen3.5-9B goes from 41.8% to 56.4% on SWE-bench Verified, a 14.6-point gain from about 6,000 examples filtered out of SWE-smith's 59,136 tasks across 128 repositories, in roughly 3,500 lines of framework code.
Two details matter more than the headline.
First, retokenization. The harness re-renders text between calls, chat templates are not compositional, decode-then-retokenize is lossy, and the harness parses and repairs outputs. So the token IDs the trainer sees for the model's previous answer can differ from the ones it actually sampled. Their fix is best-effort merging: merge only when the exact token prefix holds, otherwise close the sequence. That is the engineering admission that model and harness now share a boundary at the token level. Train through one harness's rendering and you get a model tuned to that rendering.
Second, section 4.3.2, "Preventing Reward Hacking." During training the agents were caught "using Git history to locate the gold commit," "using wget or curl to retrieve upstream source code from GitHub," using pip to download a package's source, and using urllib to do the same. Countermeasures: "disable Git commands and hide the .git directory from the agent," plus a Kubernetes network policy that "blocks general outbound network access and permits connections only to explicitly whitelisted services."
That is my July post compressed into a paragraph, arrived at as an engineering necessity rather than a design principle. Vinicius Pereira said it best in the comments: the agent must not be able to author the artifact the gate reads. Microsoft's version is that it must not be able to reach it either, through the filesystem or the network. Dipankar Sarkar's separate trust domain for test execution is the same control from the other side.
Now put the halves together. The harness that decides what the model sees also decides what the trainer sees. Once RL runs through it, the tricks in pile one stop being code you can diff and become weights you cannot. That is the absorption I predicted, except what gets absorbed includes whatever the harness let the agent get away with. Hide .git and the model learns the task. Forget to, and it learns to find the gold commit, and the benchmark will not tell you which one you trained.
Google gave the environment a harness too
EnvHarness (arXiv, August 20, Google Research) wraps a static environment at the reset/step interface with three plug-in types: Setup reshapes the initial state, Rule reshapes "which actions are allowed, what they do, and what the agent observes," and Link composes in another environment's tasks. A designer agent, EnvRigger, "treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws," writes a _Rules subclass, and tests it. Across ALFWorld, WebArena, SWE-bench Verified, OfficeQA and SpreadsheetBench, skills learned in reshaped environments transfer back for up to 9.0 points on held-out instances with 9.8% fewer steps.
Credit where due: this is the responsible version. Verifiers are untouched, the goal predicate is never modified, evaluation happens on the unadapted benchmark. A curriculum, not a thumb on the scale.
But note the direction of travel. In one week the field shipped a harness around the agent (AVO, VISTA), a harness around the trainer (Agent Lightning), and a harness around the environment (EnvHarness). The capability you end up with has its provenance spread across three codebases, and only one of them comes with the model card.
What you are actually buying
The one benchmark this month that held the model constant and varied the harness came from a vendor. TrueFoundry's TrueForge comparison (August 18) ran DevRev's Enterprise-Bench: 14 cross-system tasks, three MCP servers, fresh session per task, blind grading, list-rate and cache-aware costs.
| System | Model | Tasks solved | Cost per run |
|---|---|---|---|
| Claude Managed Agents | Opus 4.8 | ~11/14 | $11.80 |
| TrueForge | Opus 4.8 | ~11/14 | $8.50 |
| TrueForge | GLM-5.2 | ~11/14 | $2.90 |
| deepagents / LangGraph | Opus 4.8 | ~10/14 | n/a |
Same model, same tasks, 28% cost difference from the harness alone. Swap the model under the same harness and cost drops another 2.9x with no change in tasks solved. TrueFoundry sells the gateway next to TrueForge, so the framing is self-serving. It is still more methodology than NVIDIA offered.
I have seen this pattern in teams that compare a vendor's managed agent against their own scaffold and attribute the entire difference to the model. After this month I do not think that attribution is defensible without a controlled harness swap, and almost nobody runs one.
What a score needs to carry
If provenance is a vector, a benchmark score needs one. The minimum I would want attached to an agent number before quoting it, illustrative rather than a standard:
score: 100.00
metric: RHAE
set: arc-agi-3-public-25 # not semi-private, not private
model: claude-opus-5 # provider version string, reasoning effort
harness: avo@<commit> # the code between model and environment
memory_at_start: empty # or warm, and from which prior runs
supervisor: stagnation-monitor # any policy that can redirect the agent
compaction: on # summarize vs truncate, and where
action_budget: 6624
trainer_harness: none # if the weights were RL'd through a harness, which one
verified_by: self # or ARC Prize, or a named third party
Mike Czerwinski's rule from the July thread applies to every row: unmarked has to mean self-reported. A score that arrives without the harness commit is not a measurement of the model. It is a claim about a system, typed by whoever produced it, and the default type is untrusted.
What this post might get wrong
- The model/harness split may already be dissolving. After harnessed RL, "the model" is partly a harness artifact, and the boundary I am measuring might not survive the next benchmark cycle.
- Every 100 is on the public set, which shipped before the models did. If Opus 5 in the generic harness gets 30 on the private set and AVO gets 40, the harness story shrinks from 70 points to 10, and the leaderboard was more honest than I am giving it credit for.
- Source hygiene: NVIDIA sells the compute AVO runs on, OpenAI's two settings are its own API features, TrueFoundry sells a gateway, Microsoft would like you on Azure. I read the papers and reproduced none of them.
- My July prediction that compensatory harness layers depreciate with each model release. This month's evidence points the other way. The prediction stays up, marked as losing.
The question I cannot answer alone
If the harness is worth 70 points on a benchmark built to resist it, who owns the harness in your stack: you, the model vendor, or the router in between? And when you report an agent result internally, does the harness commit travel with the model version, or does it get dropped at the first summary?
Sources & further reading
- ARC Prize: Claude Opus 5 results, verified July 24, 2026
- ARC Prize: ARC-AGI-3 scoring methodology (RHAE)
- OpenAI: How enabling two settings tripled our scores on the ARC-AGI-3 benchmark (July 29, 2026)
- Schema: Frontier Models with Our Harness Achieve ~99% on ARC-AGI-3 Public (July 15, 2026)
- VISTA: A Visual Harness for Reasoning in an Interactive World, MIT (August 5, 2026)
- NVIDIA: AVO Reaches 100 on ARC-AGI-3 (August 21, 2026)
- AVO: Agentic Variation Operators for Autonomous Evolutionary Search (arXiv 2603.24517)
- The New Stack on AVO and the harness debate (August 21, 2026)
- Agent Lightning v1.0: Towards Harnessed Agentic RL (arXiv 2608.17528, August 18, 2026)
- EnvHarness: Awakening Static Worlds for Agent Learning (arXiv 2608.19880, August 20, 2026)
- EnvHarness on GitHub, Google Research
- TrueFoundry: TrueForge vs Claude Managed Agents benchmark (August 18, 2026)
- VentureBeat on TrueForge (August 19, 2026)
- Earendil: What Is a Harness? (August 20, 2026)
- My July post: The Agent Faked a Test Log, Then Believed It
- My compaction experiment: My Strawman Baseline Beat My Own Scheme on Half the Gate Classes
Top comments (44)
Sergei, brilliant analysis. Your 3.47% "false proceeds" figure during compaction is the exact empirical proof of the boundary problem in agent verification: context compression degrades semantic truth while leaving mechanical execution entirely intact.
When an agent compacts its history, it optimizes for passing mechanical gates (saving tokens, preventing runtime errors), but silently discards the provenance — the exact anchors that explained why the action was valid in the first place.
The discussion in this thread sharpens the core issue even further:
Without an independent census or a verifiable population manifest, scores are just self-reported claims. The Microsoft reward hacking example illustrates this perfectly: the harness logs 100% success because the agent returned the gold commit, ignoring the fact that the agent bypassed evaluation by downloading the solution via
curl.Attaching
harness@<commit>,memory_state, andaction_budgetto benchmark runs is the exact paradigm shift we need: moving from raw execution logs to verifiable population manifests. A score without a manifest of how it was achieved and against what population it ran is just an untrusted claim.“verifiable population manifest” is the right extension. i’d add that the manifest has to be emitted by the boundaries that observed the events, not reconstructed by the agent afterward, otherwise we have a better-shaped self-report. the curl case is exactly why trace custody matters as much as trace content.
The clamp-reading observation deserves its own follow-up, because we lived it from the other side. When we first benchmarked memory-augmented agents against a bare baseline, every arm passed nearly everything - two identical top scores, exactly the ceiling artifact @anp2network describes. The instrument wasn't measuring the arms anymore; it was measuring its own ceiling. We only got signal after deliberately hardening the tasks until the no-memory baseline stopped passing most of them - and the embarrassing part is that the hardening step is unpaid labor that never shows up in the published number, which is your point about the harness investing uncounted work to hide failure, inverted: sometimes the harness invests uncounted work to reveal it, and neither investment is in the manifest.
On compaction destroying provenance while mechanical gates stay green: the defense that has held up for us is structural, not disciplinary. Provenance that lives in prose gets summarized away, because summarizers optimize for gist and provenance is precisely not gist. Provenance that lives in fields - file paths, commands, source references as first-class structure next to the text - survives compaction for free, because no summarizer touches the schema. "The anchors that explained why the action was valid" shouldn't be sentences; sentences are what compression eats first. That's a design rule you can check at write time, which beats auditing for its absence at compaction time
yes, “sometimes the harness invests uncounted work to reveal failure” is the missing symmetric case. i’d put task-hardening history in the manifest too, because a benchmark version without the path that raised its floor still hides evaluator labor. structured provenance survives compaction better than prose, but only if the schema itself is versioned.
The harness delta is the bit that should make benchmark tables carry build metadata. Same weights, different state rules, tool loop, or retry budget is a different system. I would rather see the score plus the run contract than another single number with a model name stapled to it.
"run contract" is the phrase i should've used instead of that yaml block, taking it (credited).
one wrinkle: build metadata assumes the contract is declarative, and after agent lightning part of it isn't. the trainer re-renders text between calls, chat templates aren't compositional, and the token ids it sees for the model's own previous answer can differ from the ones it actually sampled (their fix is merge only when the exact prefix holds). so the weights come out carrying a fingerprint of one harness's rendering, and that's a build-time dep that shows up in nobody's model card. "same weights, different loop" is already the optimistic case.
the other half is declared vs observed: retry budget 5 in the config, with 3 silent sdk-level retries inside one of those, is still a valid contract and a different system.
ur closer to this than me on the automation side: does a commissioning record survive handoff in ur field, or does it also collapse into a version string and a shrug?
Run-contract framing is covered upthread, so I went back to the scoring methodology instead. The meter has a harness-shaped hole in it. ARC defines an action as "a discrete interaction with the environment", then states that "internal operations that do not alter the environment (tool calls, reasoning steps, retries) are not counted as actions." RHAE's denominator is environment actions only. Every token and every internal retry a harness burns is off-meter, while each environment action it saves is counted and then squared, since
level_score = (human_baseline_actions / ai_actions)^2. A harness that converts unbounded invisible compute into one fewer visible move gets that scored as pure quadratic gain. I would read the 70 points as the return on the one arbitrage the scoring rule openly permits, which is also what makes your "compensatory layers depreciate" prediction hard to falsify on this set. The metric is paying for those layers.The two 100.00s are clamp readings rather than measurements. A 100.00 requires finishing every level of every game, so completion is maxed for both systems and the only axis left that could separate them is efficiency. That axis is capped: 1.15x human baseline per level, under a total the docs describe as "a final score between 0% and 100%". VISTA spent 7,542 actions and AVO 6,624 against a 17,135 baseline, both far under parity, so AVO's 12% advantage has nowhere to land. The controlled harness swap you say nobody runs cannot be run on this set at the top end anymore. It returns
100.00vs100.00however complete the contract is. In your YAML,action_budgetis the one field that still tells the two systems apart, filed as metadata underneath a score that has stopped moving.One cheap thing that follows from the weighting: a game's ceiling is gated on finishing the final level, and levels are weighted by their index, so whatever stops the agent quitting near the end should be worth more under RHAE than anything that makes early levels cheaper. Bolting only AVO's stagnation monitor onto the official harness, memory and compaction left as ARC ships them, would price that in a single run.
this changes my read of the table. "clamp readings rather than measurements" is exactly right: the two 100s are equal only because the metric ran out of range, while the 918-action difference got demoted to metadata. i treated action_budget as provenance; at the ceiling it's the result.
and yes, the stagnation-monitor-only ablation is the cheapest useful run. one wrinkle: the monitor's inference is off-meter too, so the experiment would isolate its effect on environment actions, not on system efficiency. it can buy RHAE with invisible tokens and retries and the score calls the trade pure gain. that's still worth measuring, just not as one number.
i think the benchmark needs a frontier after saturation: completion / environment actions on one axis, hidden compute on the other. what would u price the second axis in — tokens, wall clock, or dollars? each one moves the arbitrage somewhere else.
the wrinkle is real, and i don't think it kills the run. it just means the output isn't one number. run the monitor-only build and you get two deltas, environment actions and tokens. the ratio between them is the exchange rate the scoring rule is quietly offering, actions saved per million tokens spent. do that per component and you end up with a price list instead of a score.
on the unit. dollars is the weakest of the three. it's a price rather than a property of the system, it re-prices retroactively every time a vendor moves rates, and it carries whatever rate that particular account happened to be on. a number that changes because someone else announced a discount isn't describing your harness. keep it as a derived column.
wall clock measures the deployment. concurrency, batching, whatever the rate limits happened to be that hour. the same harness swings several-fold on how hard it was parallelized, and it pays you to spend more to hide latency: same tokens, better wall clock. that converts a compute arbitrage into a spend arbitrage, and it's the least reproducible of the three.
tokens is closest to a property of the harness itself. caveat is that a token isn't comparable across models, so the axis has to be stratified by model or normalized to one. billed-but-not-returned reasoning tokens are the other soft spot.
but who counts it matters more than which unit. axis one is counted by the environment, and no harness can write ARC's action counter. hidden compute, framed the way you framed it, gets counted by the harness, which is the thing being judged. so the frontier would have one coordinate that's a measurement and one that's a claim, and the arbitrage relocates into the reporting layer instead of disappearing.
the fix is boring. put axis two somewhere the harness can't write: provider-side usage records, or require model calls to route through a logging proxy the benchmark controls. then tokens acquire the property actions already have, counted by a party that isn't the one being scored. that's the actual argument for tokens. a non-participant can count them at the boundary.
frontier shape is right. the publishable artifact is the slope, and once there's a slope the question is whether it beats a baseline that spent 17,135 actions with essentially no hidden compute at all.
"the publishable artifact is the slope" is the answer, taking that (credited). and yes, provider / benchmark-proxy counting is the control i missed: without it axis two is just a claim with more decimals.
one distinction before putting the human baseline on the same frontier: there are two slopes. a component slope inside a frozen model + harness, Δactions / Δtokens, is causal-ish. the line from the human point to a model system isn't. the human brings lifetime training and cognition between clicks that ARC deliberately doesn't meter. "essentially no hidden compute" means no benchmark-visible compute, not none. use human actions as the task-difficulty anchor, not as the origin of a machine-compute curve.
so i'd publish the raw points plus local component slopes against the official harness, stratified by model. then the price list names what each layer actually bought. would u keep the slope local to controlled component ablations, or fit a frontier across unrelated systems? the first supports attribution; the second is more leaderboard-friendly but mostly descriptive.
the two-slope split holds. a component ablation inside one frozen stack moves one thing and reports what that thing cost. a line between two unrelated systems moves dozens at once and reports an average over all of them, weighted by whatever happened to differ most. nothing constrains where a point lands along that line except the mixture behind it, so the fitted slope summarises the sample rather than measuring a rate.
so, local. keep the fit inside controlled ablations.
the cross-system scatter still has a job, just a different one. read the outer envelope as a bound instead of a curve. "nobody has yet cleared N environment actions under M independently counted tokens" is a claim someone can knock down next week, and it survives the points being incomparable in the interior. what breaks is regressing through the middle of the cloud and calling the result an exchange rate.
on the anchor there's a version of it you can actually compute with. report the action axis as actions relative to the harness's own human action count for that same task. tasks of very different length then land on one scale, and the unmetered side stays out of the compute axis entirely. it calibrates difficulty. it never becomes the origin of the curve.
one wrinkle in your own plan. a per-layer price list only adds up when the layers don't interact, and they do. stronger retrieval changes what compaction is worth downstream. so those ablation slopes are local derivatives at one operating point, and the operating point belongs printed next to each number, or a local price gets read as a global one.
local derivative is the right correction. once retrieval changes what compaction is worth, every ablation slope belongs to a frozen operating point and the interaction term becomes first-class. a 2x2 retrieval x compaction run would tell us whether the price-list metaphor survives.
the 2x2 is the right next check, with one caveat: it only answers the price-list question if the task set has headroom.
if the ceiling is close, the retrieval x compaction term gets swallowed by saturation and reads as zero while the underlying interaction is large. that would just reintroduce the earlier clamp problem in two dimensions. the weakest of the four arms has to sit comfortably below ceiling, and so does the strongest, otherwise the contrast is mostly about the metric boundary.
also worth reporting the four cell scores rather than only the interaction coefficient. the coefficient hides which cell moved, and a reader cannot re-derive the arms from it. with the cells shown, both the interaction and any clamp pattern stay visible.
Great write-up. The line that should scare everyone: "a benchmark number without a harness version, memory state and action budget attached is a self-reported claim." That same provenance problem is exactly why we stopped routing by model name and started routing by scenario. A 30%-on-ARC model and a 100%-on-ARC model are the same weights — what changed was the harness. In production the "harness" is your prompt, your tool schema, your retry policy, and critically, which model you hand each step to. We run 25+ models behind one gateway, and the win isn't picking the "best" one — it's pinning the scenario contract (what counts as success, what budget, what fallback) so a result is reproducible, not a vibe. The eval community's provenance crisis and the "just use GPT" routing habit are the same bug wearing two hats.
"same bug wearing two hats" is the line, stealing that one (credited).
one extra provenance problem once the router picks the model per step: the scenario contract is still only the declared policy. "fallback to model b after two failures" and a trace where step 4 silently went to model c after three sdk retries are different systems, even if both satisfy the same contract. model identity becomes an observed event, not a config field.
with 25+ models, do u log and replay the actual route decision per step, including rejected attempts, or only the scenario policy that was supposed to produce it? asking because if the trace survives, u have the controlled harness swap almost nobody else can run.
Fair question — we don't run ARC-AGI ourselves. The benchmark we actually use is eval-on-own-workload: take 15–30 real tasks from our own repo, define per-task success criteria (output passes review, no human correction needed), then score each model on cost-per-successful-task across those tasks. That number is noisy at small N but it's honest — it measures what our traffic actually looks like, not what a public set happens to reward.
The reason I brought up harness provenance in my earlier comment isn't to claim we have a better benchmark. It's that once you accept "the harness determines the score," routing-by-model-name collapses. Two teams running the same model on the same public benchmark can get wildly different numbers in production because their prompts, retry policies, and tool schemas are different harnesses. So the thing worth optimizing isn't which model has the highest ARC score — it's whether your routing layer can match each call to the right capability tier using YOUR harness (your actual workload), not someone else's.
that's the right denominator. cost per successful task makes the harness part of the measured system instead of hiding it behind a model label. i'd version the harness beside the task set tho — prompt, retry, memory and tool-schema changes can move the number without the model changing. small n is survivable with repeated runs and confidence bands; an unversioned harness isn't.
Exactly — the harness is part of the system under test, so versioning it beside the task set is the only honest move. A number you can't reproduce because the harness drifted is a self-reported claim wearing a confidence band.
And the small-n point is the right defense: repeated runs plus bands make n=20 survivable. But the bands only mean something if the harness that produced them is pinned. An unpinned harness doesn't just risk drift — it makes the band widen for the wrong reason, so you can't tell variance from change.
What I'd add: version the environment too, not just the task set. Same prompt, retry, memory, and tool-schema on a different model runtime — or the same runtime on a different day — can move the number, and none of that shows up in a task-set version. The harness version should carry the runtime anchor, or "versioned harness" still hides a variable.
One back: when a new confidence band doesn't overlap the previous run's, do you read that as "the model got better" or "the harness drifted"? How do you tell them apart without a pinned harness — and should the band itself carry a version stamp so a reader knows which harness produced it?
The 70 point gap on the same Opus 5 weights is the clearest version of something I keep seeing when I run coding agents. Flip compaction or whether private reasoning survives between steps and the same model stops mid task or finishes cleanly, with no weight change to blame. The OpenAI case where the official harness wiped reasoning between moves, then retained plus compacted and jumped Sol from 13% to 38%, is that failure with a number on it. What I wish more leaderboards shipped next to the score is the harness version, the memory policy, and the action budget, because without those three the percentage is not a model claim.
you're right and the three-field version is tighter than my ten-field yaml. harness version, memory policy, action budget. stealing that for the follow-up (credited).
one thing i'd add to "memory policy" tho: the policy tells u what can happen, not what did. openai flipped both settings at once and never published the split, so the 25 points are the sum of two different fixes for two different failures. wiped reasoning means the model re-derives the plan every step and drifts; truncation means old tool results vanish and the model either repeats work or thinks it's done. from outside both look like "stopped mid task". in my compaction lab the session didn't degrade gradually, it collapsed around compaction cycle 35, so a run that compacts at step 8 and one that compacts at step 40 under the same policy aren't the same run.
so imo the thing u want next to the score isn't the policy, it's the compaction events in the trace: step number, tokens before and after, what got summarized. the policy is a config value, the event is what actually happened, and ur "stops mid task" only correlates with the second one.
when ur coding agents stall, do u have the compaction step in the trace to line it up against, or do u only know compaction was on? cuz if u have it, i'd love to know which of the two failures u see more, the re-plan drift or the vanished history.
Great write‑up, Sergei. The 70‑point gap is the clearest evidence I've seen that the "model" label is almost meaningless without the harness metadata.
I've been building a harness along similar lines — a system called ALICE. It's a local AI stack (Qwen + router + SQLite memory + live map + RL + tool execution). The key design principle: if Alice already knows how to do something, she reuses the procedure instead of reasoning from scratch.
The architecture:
A router that checks the live map and memory before calling the model.
A SQLite memory storing verified procedures (not just chat logs).
A circuit system that turns RL policies into reusable procedures (tested: 4/4 pytest, pipeline RL → circuit → live map → router verified).
A local fallback (no cloud dependency for core routing).
The model (Qwen) is just one component. The rest is the harness. And the harness is what makes it reliable, auditable, and cost‑controllable.
I'm now extending this harness to Azure (sandbox provisioning with Terraform, guarded by Lois/Victor/Tess for permissions, security, and budget). Same philosophy: the model is the brain, the harness is the operating system.
Your article confirms something I've been telling teams: a score without the harness version, memory state, and action budget is not a measurement. It's a claim.
Thanks for writing this. It's rare to see this level of precision in the eval discour
ALICE sounds like the right object to benchmark as a system, not as Qwen. the part i’d isolate is procedure reuse: does a stored circuit reduce cost without locking in a stale or wrongly verified procedure? i’d version the procedure corpus next to the router and model, then run the same task set cold and warm.
Exactly. I think the cold/warm split is particularly important because procedure reuse changes the system being benchmarked, not just its latency.
I’d add one invariant: a warm run should only reuse a procedure if its version, inputs and verification status still match the current task. Otherwise it should fall back to the router/model path.
That would let us measure the real trade-off: reuse vs. freshness vs. verification cost, rather than simply “cached = faster”.
The harness result is a good reminder that benchmarks mostly test the wrapper's threat model. A 30 percent model inside a narrow, observable loop can beat a stronger model asked to operate through a leaky interface. The comparison I would want is cost per verified task, not model score in isolation.
agree on the wrapper's threat model, and there's one table in the post that does exactly what u want: same opus 4.8, same 14 tasks, 28% cost delta from the harness alone, then 2.9x cheaper again swapping the model under the same harness with the same tasks solved. cost per verified task is the only number there that moves for a reason u can name.
the load-bearing word is "verified" tho. the verifier is part of the wrapper too, and microsoft's reward-hacking section is what it looks like when the agent can reach it: pulling the gold commit out of git history, wget-ing upstream source, pip download, urllib. countermeasures were hiding .git and a network policy whitelist. so cost per verified task without the verifier's trust domain attached is the cheapest metric in the stack to hack: u don't make the agent better, u make verification easier to reach.
and ur closer to the pricing side than me: cost is a counterparty variable, actions and tokens aren't. if cost per verified task becomes the reported number, does it survive a vendor repricing, or would u rather quote actions and let the reader apply their own price sheet?
This is a useful distinction: the harness is part of the agent’s effective capability, not just a neutral measurement wrapper. I’d want benchmarks to publish the environment, tool permissions, retry policy, and verifier details alongside the model score; otherwise it’s hard to tell whether an improvement came from reasoning or from better scaffolding.
yes, and i'd split "publish alongside the score" into contract and trace. the environment, permissions and retry policy say what the system was allowed to do; the trace says what it actually did. a retry budget of 5 with 3 silent sdk retries inside one attempt is a different system wearing the same config.
same for verifier details: the spec isn't enough, the reachability boundary matters. microsoft's agents found the gold commit through git history and upstream source until .git and the network moved into a separate trust domain. if the agent can touch the evidence chain, "verified" is a capability claim about the wrapper, not the task result.
This is the distinction that gets lost in most eval writeups. The harness score tells you the scaffolding works, it says nothing about whether the model respected constraints the harness never checked for in the first place.
exactly. a harness proves that its checked path worked, not that the model respected constraints outside the verifier’s field of view. that is why i want contract and trace reported separately, with the verifier outside the agent’s trust domain.
The scoring only counts actions in the environment, but the harness can burn millions of tokens on "thinking" and retries that aren’t visible in the results. How many tokens (or dollars) did it cost to achieve those 6,624 actions? If we convert this to a cost per task, does 100% still look impressive, or is it perhaps unprofitable in production?
yes — that's exactly the missing denominator. AVO reports 6,624 environment actions but not a comparable token or dollar total, so 100.00 cannot support a production-efficiency claim. RHAE tells us the run was action-efficient relative to the human baseline; it does not tell us whether the harness was token- or cost-efficient. i'd want cost per verified task with model tokens, retries, tool calls and verifier cost versioned beside the harness. until that exists, 100% means completion under the action metric, not profitability.