The Only Coding-Agent Benchmark That Matters Is Your Own Revert Rate

• Book: System Design Pocket Guide: Fundamentals
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

Meta's Muse Wonder model card, dated October 12, 2026, has a line in its benchmark notes that's easy to miss. On SWE-Marathon, one of its coding rows, Abundant AI's leaderboard results for GPT-6 Astra and Claude Fable 5.1 came from runs in Codex and Claude Code, "so this row is not a controlled common-harness comparison."

That is an honest note. It also sums up the trouble with coding-agent leaderboards in one line. Each number belongs to a model running in someone else's harness, on tasks someone else picked. Your repo is in none of them.

You already have a better scoreboard: your git history. It records every change that got reverted, needed a fix within a week, or turned main red. It just doesn't know which agent wrote what. That part takes one line per commit.

The leaderboard is scoring the harness too

The same model card says it outright in its methodology: "External coding results retain the harness or agent product used by the reporting vendor or model provider." For the runs Meta did itself, competitor models used common settings "where practical but remain best-effort: prompts, tools, and runtime may not be optimally tuned for proprietary models."

Epoch AI, which runs SWE-bench Verified independently, logged the same effect from the other side. Its changelog entry for February 12, 2026 reads: "We performed a major upgrade of scaffolding, environments, and token limits." The next line: "This led to model performance improving significantly." The benchmark kept its name while the setup around the models changed, and the scores went up with it.

That isn't cheating, but it means a leaderboard ranks a model together with its setup, on tasks that end when the tests pass. You run a different combination on different tasks, and your tasks don't end when the tests pass. They end when nobody has had to touch the change again.

DORA measured the part leaderboards skip

The 2025 DORA report, State of AI-assisted Software Development, surveyed nearly 5,000 technology professionals, and 90% said they use AI at work. Google's announcement puts the headline finding in two sentences: "Unlike last year, we observe a positive relationship between AI adoption on both software delivery throughput and product performance." Then: "However, AI adoption does continue to have a negative relationship with software delivery stability."

Stability has a precise meaning in DORA's model. Its metrics guide groups two metrics under instability. Change fail rate is "the ratio of deployments that require immediate intervention following a deployment," likely a rollback or a hotfix. Deployment rework rate is "the ratio of deployments that are unplanned but happen as a result of an incident in production."

There won't be a 2026 survey to update the picture: on September 3, DORA said it won't run its annual survey or publish a 2026 edition of the report (LeadDev). So the 2025 finding is still the latest one: more throughput, less stability. Public benchmarks measure the first half. Your history records the second half, change by change. A revert is a rollback that git wrote down. A fix to a change that landed four days ago is rework.

Medals from someone else's tasks, against a tally from your own history.

Tag every landed commit with the agent and the model

Some agents already sign their commits, each in its own way. Claude Code adds a trailer to commits that you can change or hide with its attribution.commit setting. Aider appends "(aider)" to the git author and committer names when it wrote the change. What you want is one format that names the agent and the model, whichever vendor you run.

Git already has the format. A trailer is a Key: value line in the last paragraph of a commit message, after a blank line. git commit --trailer adds one for you. When you land a task branch as one squashed commit:

git switch main
git merge --squash hermes/fix-login
git commit -m "Fix flaky login test" \
  --trailer "Agent: claude-code" \
  --trailer "Model: claude-opus-5-5"

If the commit already exists and you haven't pushed it, amend it:

git commit --amend --no-edit \
  --trailer "Agent: claude-code" \
  --trailer "Model: claude-opus-5-5"

To check that git reads them as trailers, pipe the message through git log -1 --format=%B | git interpret-trailers --parse. It prints only the trailer lines. If a task moved from one agent to another halfway, write two Agent: lines.

The second signal needs a convention for fixes. The Linux kernel's patch guidelines already have one: when a patch fixes a bug from a specific commit, add a Fixes: tag "with at least the first 12 characters of the SHA-1 ID, and the one line summary." Use it whenever you repair something that landed recently, whoever wrote it.

Count what happened after landing

This script reads the last 90 days of git log, finds the commits with an Agent: trailer, and counts three things per agent and model: reverts, Fixes: commits within 7 days, and failed CI runs on main (from gh run list). Trailers come from git's own parser through the %(trailers) placeholder.

#!/usr/bin/env python3
# agent_scores.py: what happened to each agent's landed work
import json
import re
import subprocess
from collections import Counter

WEEK = 7 * 86400
FMT = ("%H%x1f%ct%x1f"
       "%(trailers:key=Agent,valueonly,separator=%x2C)%x1f"
       "%(trailers:key=Model,valueonly,separator=%x2C)%x1f"
       "%B%x1e")
REVERT = re.compile(r"This reverts commit ([0-9a-f]{40})")
FIXES = re.compile(r"^Fixes: ([0-9a-f]{7,40})", re.M)

def run(*cmd):
    return subprocess.run(cmd, capture_output=True,
                          text=True, check=True).stdout

commits = {}  # sha -> (commit time, "agent / model", message)
log = run("git", "log", "--no-merges", "--since=90 days ago",
          f"--format={FMT}")
for rec in log.split("\x1e"):
    if not rec.strip():
        continue
    sha, ct, agent, model, msg = rec.lstrip("\n").split("\x1f", 4)
    who = f"{agent} / {model or '?'}" if agent else ""
    commits[sha] = (int(ct), who, msg)

This first half builds a map of every non-merge commit from the last 90 days, keyed by SHA. The second half goes in the same agent_scores.py, right below it, and does the scoring and the printing:

def lookup(prefix):
    hits = [s for s in commits if s.startswith(prefix)]
    return hits[0] if len(hits) == 1 else None

reverted, fixed = set(), set()
for sha, (ct, _, msg) in commits.items():
    for m in REVERT.finditer(msg):
        if m.group(1) in commits:
            reverted.add(m.group(1))
    for m in FIXES.finditer(msg):
        target = lookup(m.group(1))
        if target and 0 <= ct - commits[target][0] <= WEEK:
            fixed.add(target)

runs = json.loads(run("gh", "run", "list", "--branch", "main",
                      "--status", "failure", "--limit", "500",
                      "--json", "headSha"))
red = {r["headSha"] for r in runs} & set(commits)

def tally(shas):
    return Counter(commits[s][1] for s in shas if commits[s][1])

landed = tally(commits)
rv, fx, ci = tally(reverted), tally(fixed), tally(red)
bad = tally(reverted | fixed | red)
print(f"{'agent / model':34}{'landed':>7}{'revert':>7}"
      f"{'fix<7d':>7}{'red CI':>7}{'any':>7}")
for who, n in landed.most_common():
    print(f"{who:34}{n:7}{rv[who]:7}{fx[who]:7}{ci[who]:7}"
          f"{bad[who] / n:7.0%}")

Run it from the repo root with python3 agent_scores.py, with gh signed in. The last column is the share of an agent's landed commits that hit at least one of the three. I haven't run this against a real team's history, so read it before you trust it, and adjust it to how your team lands work.

It assumes three things. Each task lands on main as one squashed or rebased commit carrying the trailers. Reverts use git's default message: git revert starts the body with "This reverts" and the full commit ID, unless you turned on revert.reference, which switches to a short reference format. And CI runs on pushes to main. If your CI only runs on pull requests, the red column stays at zero, because the PR head commit isn't the squashed commit on main.

What the number won't tell you

Small counts lie. Three reverts out of twelve landed changes could easily be noise. Give each agent a few dozen landed changes before you read anything into the gap.

The task mix skews it. If one agent gets the schema migrations and another gets the copy changes, the first one will look worse every time. A third trailer, Task: migration, or grouping by the top-level folder a commit touched, lets you compare agents on the same kind of work.

Not every revert is an agent's fault. A product call, a feature flag that went the wrong way, a flaky test that failed once on main: all of those show up in the table. Read the revert messages for the agents near the top before you change anything.

An undo that rewrites history before you push, whether git reset or a tool's undo button, leaves no Revert commit behind. Those changes vanish from the count. If you want them in, write them down.

And it's slow. You need weeks of history before the table says much, so a leaderboard is still a fine way to build the shortlist. Your history decides what stays on it.

Make the table something you can act on

I build Hermes IDE, so weigh this section with that in mind. A revert-rate table is only useful if changing agents per task is cheap. In Hermes, each task picks its agent, model, effort and account, and you can save the combinations you use often as presets. If the table says one agent and model pair keeps getting its migrations reverted, the change is a different preset for that kind of task.

The ⌘N New task sheet in Hermes 2.3.4: the task typed, with chips for Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort, and a preview of the claude command Hermes will run.

Each task also starts on its own hermes/<name> branch in its own worktree, so one task is one branch and, once squashed, one commit with one set of trailers. Before that commit exists, the Review Desk (⌘G) shows what the agent changed, turn by turn or file by file. The Land action commits the work, then opens a pull request or merges locally into the branch the task started from, and you can undo it. If you undo a local Land before you push and it leaves no Revert commit, it falls out of the count like the undos in the last section, so write that one down too.

The Review Desk (⌘G) in By turn view in Hermes 2.3.4: two turns of a Claude Code task in orders-api, turn 1's diff of src/db.ts, the changed files, and a commit message drafted from the turns.

Hermes doesn't document a commit-trailer option, so the Agent: line stays a git step: amend before you push, or put it in the squash commit message of the pull request. When one agent hits its usage limit, Hermes lets you continue the task in another, which is exactly the case for two Agent: lines.

Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It's not affiliated with Nous Research, Anthropic, OpenAI or any agent vendor. The tagging and the script work the same without it.

If you tagged the last three months of your history, which agent would you expect at the top of the revert column, and would you bet on it?

If this was useful

A revert is a recovery step, and change fail rate is an availability number long before it's an agent number. The System Design Pocket Guide: Fundamentals covers the building blocks that decide how much a bad change costs you once it ships: availability, failure and recovery, caching, rate limits and load balancing. If you want each task to pick its own agent and model, that's what I build Hermes IDE for, and the repo is open for issues.