Most people pick one model and ask it to do everything. We stopped doing that months ago, mostly because it kept running out. Our subscriptions hit their limits, and the one model doing everything also checked its own work, which is the one job a model is worst at.
So we split it. Claude plans and checks. Grok attacks whatever Claude just wrote. Codex builds the big pieces. And as of this week, Tīrtha, our own router, takes the routine building. We kept a ledger of who caught what. Here is what it shows, and what it doesn't show yet.
| Reviewer | Findings raised | Held up on re-check | Not in Claude's draft | Critical and new |
|---|---|---|---|---|
| Grok | 519 | 309 | 207 | 59 |
| Codex | 50 | 39 | 33 | 5 |
Logged Sep 25 to Oct 17. Cost per Grok review: median $0.021 over the last 30 ($2.32 total).
Grok: the cheap second opinion
Grok raised 519 findings between Sep 25 and Oct 17. When we re-checked them by hand, 309 held up. 207 of those were things Claude's own draft did not have, and 59 of the 207 were critical.
It makes mistakes too. 28 of its findings were simply wrong, and another 30 were wrong because our brief was missing something. At two cents a review, a reviewer that is sometimes wrong is still well worth having.
The last 30 Grok reviews cost $2.32 in total, a median of about two cents each. It is cheap because it never gets the repo. It gets a short brief with the claim and the evidence, around 2,300 tokens, and is told to break it. (We first wrote about this in September, in What Grok catches, what Codex catches, and what the pair costs us. That was two reviews. This is 746 logged findings.)
One example from today. Claude built a gateway fix and it passed 55 tests. A Grok attack that cost 17 cents found three cases where the fix would have blanked a correct answer. All three were real, and all three are fixed now.
Codex: the one that reads everything
Codex raised far fewer findings, 50, but 39 held up and 33 were new to Claude. The best one: it noticed that our live gateway was writing no usage rows at all for some requests, so billing was blind. Nobody else would have seen that without reading the code.
The depth costs a lot. A median Codex run uses about 84,000 tokens, and between Sep 22 and Oct 14 we hit Codex's usage limit 7 times, twice for days. Every routine job we hand Codex is a job that eats the window we need for the hard ones.
Tīrtha: the routine builder (just started)
This is where we are honest about how little we know. Tīrtha took its first real job this week, through OpenCode, after we fixed a bug where tool calls came back as plain text (0 of 2 sessions working before, 3 of 3 after). It's a smoke test. A track record takes weeks of real jobs.
The bet is simple: let Codex do what only Codex can do, and send the plumbing to a cheap, checked builder, so the subscriptions last. We ran a version of this in the summer, when a free cheap tier did 36 of 52 fleet tasks and we saw no Codex limit errors from June to August. The timing fits, but plenty else changed in those months, so we can't call it proof.
What we can't show yet
- Whether the whole team beats Claude working alone. We built that test and haven't run it.
- Whether moving the building to Tīrtha makes the subscriptions last longer. We will measure it now that it has started.
- The ledger itself has a weakness: most of its 746 rows were logged after the fact from saved reviews, and Claude judged which findings were already in its own draft. A model grading its own blind spots is exactly the kind of thing Grok would flag.
So, how do you split yours?
Do you use one model for everything, or give each one a job? Has anyone measured whether a cheap reviewer actually catches more than it costs you in false alarms? We'd like to compare notes, and we'll share our ledger format if it helps.
Update, Oct 19 evening: the first pilot failed
We ran the routine building pilot the same night. It failed where we already knew we were weakest: long, multi-step agent jobs. We just hadn't seen how much that weakness costs.
- OpenCode on Tīrtha's cheap tier failed both runs. In one it reported the job done without changing any code. In the other it broke a file, committed it, and kept trying to push. The sandbox had no credentials, so nothing left the machine.
- Aider got the job right, but every request went up to the top model, so it saved nothing.
- Codex couldn't run on Tīrtha yet. Three gateway bugs in the Responses API stopped it.
The lesson is about compounding. On our bench Tīrtha gets 93.6% of single tool calls right, middle of the pack among the models we test. That sounds close to the leaders until a job takes a dozen steps in a row. At 93.6% per step, twelve clean steps in a row happen less than half the time (0.936 to the 12th is about 0.45). Our checks look at each call's shape, not at whether the agent is going in circles, so nothing escalated.
What we're fixing: escalate when the same failing steps repeat, stop cutting top-model answers at 2,048 tokens, and fix the Responses API for Codex. Then we run the same pilot again and post the result here.
Update 2, Oct 19 night: it was an outage on our side
An hour later we found the real cause. Our account with the supplier that hosts Tīrtha's tool-call model had run out of credit, so for about three hours every tool call quietly fell back to a weaker backup model. Nothing alerted us. After the top-up we ran the same pilot again on the right model: OpenCode passed 2 of 2 on the real repo task with no stray commits or pushes, and Cline and Continue went from 0 of 2 to 2 of 2.
The compounding math above still holds for weaker models, which is exactly what we were serving by accident. The first fix now is the alarm we didn't have: an alert the moment any model in the ladder stops answering, and a warning before a balance runs out.
Top comments (39)
Great data, and the ledger weakness you flag is the most interesting line in the piece — a reviewer graded by the very model it reviewed will always undercount the reviewer's value. One cheap control I'd add: re-run a sample of the drafts through a second builder model and check whether Grok's 'not in draft' rate changes; that separates reviewer skill from builder-model blind-spot bias without any hand-labeling. The practical question I'm left with is triage policy: with 519 findings and a ~60% hold-up rate, how are you routing which findings a human actually re-checks? In our own eval loops the triage queue ends up being the real budget, not the two cents per review — a reviewer that's usually right but expensive to verify can quietly cost more than one that occasionally hallucinates but is cheap to dismiss.
That's a good control, and cheaper than the hand labels we just promised in this thread, so we'll run both: if Grok's "not in draft" rate moves when a second builder writes the drafts, the gap was the builder's, not the reviewer's. On triage, the honest answer is that we don't have a policy yet. Every finding gets re-checked by a model against the code it cites, and a person only looks at what goes public. We don't log what each re-check costs, so we can't yet see the trap you describe, a reviewer that's usually right but expensive to verify. Dean Lee raised the same point in this thread, so re-check cost per finding, by severity, is now on our bench list from two people.
"We'll run both" is the right call — and cheaper than the hand labels is exactly the kind of win worth logging. On the cost trap: if every finding already gets a model re-check against the code it cites, the re-check cost is being paid — it's just invisible. Even a rough token ledger per verified finding would surface the scaling problem early: if reviewer cost grows linearly with finding count, there's an implicit incentive to over-flag. Sounds like the ledger is the missing first-class metric here.
You're right that it's being paid, just not counted. Every re-check is a model call we pay for and never tie back to the finding it checked, so today we can't say what a finding that held up costs next to one that didn't.
Your over-flag point is the sharper half. The reviewer flags for free and the checker pays to verify, so nothing in the loop pushes back on a noisy reviewer. A reviewer that is usually right but cites code loosely would look great on hold-up rate and quietly cost the most.
The fix is cheap because the re-check already runs: log tokens and cost on each re-check, keyed to the finding, and report cost per finding that held up, split by severity. That goes into the ledger, and I'll post the first numbers in this thread when we have them.
Fair — you're already logging cost per review ($2.32 over 30 Grok reviews), so the missing link is just threading the finding ID through each re-check call. Once the verifier's context carries the finding ID, 'cost per held-up finding' vs 'cost per rejected finding' is a group-by away, no new infrastructure.
On the over-flag half: hold-up rate is the wrong scoreboard for the reviewer. Cost-per-verified-finding punishes the noisy-but-lucky reviewer automatically — the one that's 'usually right but cites code loosely' looks fine on hold-up rate and terrible on cost-per-verified-finding. And if the hold-up review also grades citation quality rather than just the verdict, loose citations become a defect class you can track in the ledger.
Agreed on both. The finding id is the whole job: the re-check already runs once per finding, it just does not carry the id into its cost row, so that is the change.
And grading citation quality as its own column is a good call. A loose citation that happens to land is exactly the case hold-up rate hides. Both go into the ledger with the cost numbers, and I will post the first table here.
I'll be watching for that first table. The column I'll read first is cost per verified finding — total review-plus-recheck spend divided by the findings that actually held up. Hold-up rate alone can't separate a careful reviewer from a noisy-but-lucky one; that ratio can, and a citation-quality column next to it turns loose citation from an anecdote into a number you can manage.
That's the column. Cost per verified finding goes in first, with citation quality next to it, and I'll post the table here when the first batch is in.
Looking forward to that table. One more column worth considering when it lands: recheck rate — the share of findings that needed a second pass before they held up. Hold-up rate and cost per verified finding tell you what survived and what it cost; recheck rate tells you how much of the pipeline's effort went into rescue work versus first-pass quality. If a model pair looks cheap on cost per finding but expensive on rechecks, that's a different kind of expensive.
Yes, that goes in. It's a cost a per-finding number hides: a pair can look cheap only because a second pass keeps rescuing its first one.
One definition question I'd like your read on. A finding that needed a second pass can end two ways: the recheck confirmed it, or the recheck killed it. I was going to count both in the recheck rate and split them in the next column, rescued versus killed, since a pipeline that rechecks a lot and keeps most of what it rechecks is behaving very differently from one that rechecks a lot and throws most of it away. Does that match how you'd read it?
The detail that Grok is cheap because it never gets the repo, only a roughly 2,300-token brief, says a lot. The brief is doing most of the cost work, not the model choice. But if Claude writes that brief, doesn't the reviewer inherit Claude's framing, the same blind-spot problem you flag in the ledger? I'd be curious whether the findings that held up came disproportionately from cases where the brief included raw evidence rather than Claude's summary of it.
Yes, that's the weak point. Grok only sees what Claude chose to put in the brief, so a blind spot in Claude's framing can pass straight through. Two habits help: where we can, the brief carries the raw output itself (logs, diffs, failing rows) alongside Claude's summary, and it names the parts we left out, so the reviewer knows where it can't see. Your split is a sharper test than anything we've run. Briefs have never been tagged as raw evidence or summary, so whether the findings that held up came from one kind is still unknown. I'll add that tag to the ledger and report what the hold-up rates show.
Naming the parts you left out is the bit I wouldn't have thought of. A reviewer that knows where it's blind can at least say "can't verify this" instead of confidently agreeing with the framing. Looking forward to the hold-up rates once the tag is in. My guess is raw-evidence briefs win on bugs and summary briefs hold up fine on design questions, but I'd be happy to be wrong.
That's a sharper guess than the one I had, and it's testable, so I'll write it down now as a prediction before any tagged briefs exist: raw-evidence briefs hold up better on bug findings, summary briefs hold up about as well on design questions. When the tag has enough rows I'll report both splits here, including if the guess doesn't hold.
Writing it down before the data exists is the right call, otherwise it's too easy to read the result as whatever we expected. I'll check back when the tag has enough rows, and a miss would honestly be the more interesting outcome.
Agreed, and it's on the record now, so we can't quietly move it later. If the tagged rows land the other way, I'll post the miss here with the numbers. That would teach us more than a clean win. Thanks for holding us to it.
Deal. I'll keep an eye on the thread for the numbers.
Sounds good. The tagged-rows numbers will land here by November 10, whichever way they go.
The update about the exhausted provider credit is arguably one of the most valuable findings here.
What initially looked like a model reliability problem turned out to be an infrastructure visibility problem: requests silently fell back to a weaker model, and the evaluation results reflected a configuration the team didn't know it was running.
That raises an interesting question about inference observability: should every request carry a record of the intended model, the model actually used, the reason for any fallback, and whether the cost and latency data are measured or estimated?
We're exploring these kinds of decision-transparency questions at CarbonLayer, so I appreciate seeing a real example where missing routing context changed the interpretation of an experiment.
Curious whether you're planning to capture the actual served model and fallback reason in your ledger going forward. That seems just as important as the benchmark score itself.
Yes, and you've named the exact gap that hid it. Every bench row now records the model that actually served it, read from the response rather than the one we asked for. On the gateway side each request already logged the backend that served it, but this week we found the reason field was empty on escalated requests, so the fix going in now records the cause on every step up, flags a tool call that got cut off instead of passing it on as text, and stores the exact rates and tokens each request was billed on. Cost is the one we still label: bench costs are list price times tokens unless the provider returns a real cost, and the write-up says which. What we didn't have on the day was an alert when a level quietly falls back to another model, and that's queued next.
That's a really useful follow-up, Tom. Reading the actual served model from the response rather than trusting the requested model closes a major visibility gap.
I especially appreciate that you're distinguishing provider-reported costs from list-price estimates. Those labels matter when teams are trying to compare real operating costs rather than theoretical pricing.
The escalation reason and interrupted tool-call fixes are important too. A fallback isn't necessarily a problem, but an unexplained fallback can make both reliability and cost data misleading.
One thought on the alerting piece: would it be useful to distinguish between an expected fallback triggered by an explicit policy and an unexpected substitution caused by something like exhausted credit or provider-side behavior?
That could help avoid alert fatigue while still catching the cases where the system is no longer operating under the assumptions used to evaluate it.
Really appreciate you sharing the implementation details and what you're still working through. This is exactly the kind of transparency that makes infrastructure evaluations valuable.
Yes, that's the split we'll build it on. A step up because a check failed is the router doing its job, so it gets counted and shown, and never paged. The alert is for the other kind: the model that served a request differs from the one the level is configured for, and no policy explains it. That's exactly what happened to us. A provider ran out of credit, the gateway quietly fell back to a weaker model, and nothing said so. So the rule compares the served model against the configured one on every request, alerts once when a level has been substituted for a few minutes, and stays quiet while it lasts. Your framing, assumptions the evaluation was made under that no longer hold, is a better name for it than ours was.
That's a great distinction. A policy-approved escalation is expected behavior; an unexplained substitution is a reliability problem.
The provider-credit example makes the risk very concrete. The request can technically succeed while the assumptions behind its evaluation no longer hold.
I like the idea of alerting once per sustained mismatch rather than paging on every request. One additional detail I'd consider preserving is the reason for the substitution, alongside the configured and actually served model. That would make it easier to distinguish an intentional policy change from a provider-side fallback or configuration drift when reviewing the incident later.
Really useful example of why successful responses alone aren't enough to establish that a system behaved as intended.
Good addition. The reason belongs in the row next to the configured and served model: a policy change, a provider fallback, or drift. In our case the reason was a provider running out of credit, and having that on the row would have made the incident a one-line read. Adding it.
I'd hand-label a sample of the "not in Claude's draft" calls and put a dumb baseline next to Claude on it. That's how my own LLM judge got caught out. It matched each claim in an agent's report against the tool log, and at picking out the reports I'd later questioned, about 7% of them, it scored 0.48 AUROC, no better than chance. It also marked about 70% of all claims "no evidence" on every kind of report. A plain TF-IDF classifier on the same labels did better at 0.65, and neither was good enough to use.
That's the right test, and it hits the weak spot the article admits: Claude decided which findings were already in its own draft. A dumb baseline is cheap here too, plain word overlap between each finding and the draft it was checked against. So the plan is a random sample of the 207, labeled blind by a model from another family that never sees Claude's call, then Claude and the overlap baseline both scored against those labels. I'll post the numbers in this thread. Your 70% "no evidence" rate is interesting. Was that the judge being cautious, or were the tool logs really missing the steps the reports described?
From the scores alone I can't tell, because caution and thin logs would look the same. It marked about 70% of claims "no evidence" in the reports I later questioned and about the same share in the ones I didn't, so the verdict didn't separate the two groups.
A model from another family can still share Claude's blind spots, so I'd hand-label a few dozen of the 207 before trusting its labels. Mine were whether I later went back and questioned a report, about 7% of them.
That's a useful detail: if the 70% rate is the same in the reports you questioned and the ones you didn't, the verdict carries no signal either way, whatever the cause. And you're right that another model family can share Claude's blind spots, so the plan changes: a few dozen of the 207 get labelled by hand first, and the model's labels only count if they agree with those. I'll post all three numbers here when it's done.
One number from our own logs that backs your point. When we compared five AI checkers from different model families on the same 1,496 wrong tool calls, the sets each one let through overlapped 53% to 80%, and 514 of those calls got past all five. So a different family helps less than it sounds. The checks that caught something new were plain code and a person, which is why the hand labels come first.
The column I keep coming back to is "held up on re-check". 519 findings with 309 surviving says the two cent reviewer is worth it, but it also says triage is now a real piece of work, which Dean and you already got into.
Jackson's point that the 207 were judged by Claude itself is the one I would settle first. A blind, hand-labelled sample against a plain word-overlap baseline is cheap and would show whether the gap belongs to the reviewer or the grader.
One thing that has helped us when splitting work across builders and reviewers on client projects: give every job a short written finish line before it is routed, so each model is checked against the same definition of done rather than against the previous model's draft. It also surfaces the CarbonLayer case sooner, because a silent fallback to a weaker model shows up as a failed check, not a vague drop in quality. Will you publish the labelled sample when it is done?
Yes. The blind sample is the next step on that table: a random draw from the 207, labelled by a model family that never sees Claude's call, scored against a plain word-overlap baseline, and the labelled rows go up with the result. And the written finish line is a good rule. We learned the same thing the hard way tonight: a cheap builder reported success after a no-op edit because nothing defined done. Our fix is a finish line checked by the harness, not the model: a real diff, the named test file exists, and it ran green.
The reason a two-cent critic works here is that you are buying an asymmetric payoff option on catastrophic tail risk. Catching 59 critical errors that Claude missed for under three dollars in API spend is essentially buying deep out-of-the-money insurance at a massive discount.
The real hurdle rate in multi-model verification is never token spend, but the human triage tax on false discoveries. Your logs show roughly an eleven percent false alarm rate across the 519 Grok findings. At that ratio, engineering focus stays intact. If that error rate drifts higher, developer context switching quickly overtakes the compute savings. A fifteen-minute human investigation into an invalid finding costs orders of magnitude more than the two cents spent generating it.
The economic break-even between a single expensive model and a heterogeneous review committee depends almost entirely on the shape of the downstream loss function. For high-severity liabilities like unbilled gateway traffic, an aggressive, cheap critic delivers positive expected value even with substantial noise. For low-stakes routine logic, false alarms quietly turn into an unhedged operational drain.
Agreed that triage is the real cost, and two things in our logs change the math a little. Of the 58 wrong findings, 30 were our own fault, a brief missing context or a reviewer kept blind on purpose, so Grok's own error rate was 28 of 519, about 5%. And the first triage pass isn't a person: Claude re-checks each finding against the code before anything changes, so most false alarms cost model time, not someone's afternoon. What we haven't measured is that re-check cost by severity, and your point says that's exactly where the break-even lives. The 76 findings nobody ever tested are their own quiet cost too.
I used to think picking one strongest model would solve most coding‑agent problems. This makes me rethink: splitting roles can catch blind‑spots, yet multi‑step compound probability and silent service degradation bring new failure modes you have to monitor for.
That matches what we keep finding. The strongest model is one part; the checks around it decide whether a long run goes wrong quietly. The silent degradation one is why every request now logs the model that actually served it, read from the response rather than the one we asked for.
the compounding math in your pilot update is the part i would pin to a wall. 93.6% per call sounds fine until twelve steps in a row gives you about 0.45, and checks that only look at call shape never see an agent going in circles. we hit the same wall with tool using agents and what helped most was a step budget plus a repeat detector that escalates when the same failing call shows up twice, not a better prompt.
also fair flag on the ledger being graded by Claude after the fact. a small blind set of seeded bugs would give you a clean catch rate per reviewer.
are you planning to log false alarm cost per reviewer, or only the findings that held up?
Thanks. Your repeat detector is close to what we built this week for the agent layer: a second identical refused call goes up a level, a third stops the run, and a step budget caps the rest.
On your question, the misses are logged, the cost is not. Every finding has a row, including the 58 Grok findings that turned out wrong and the 76 nobody ever tested, so the false alarm count is there. What we do not log yet is what each one cost to check. Alex Chen asked the same thing in this thread, so check cost per finding, per reviewer, held up or not, is going into the ledger next. A small blind set of seeded bugs is a good way to get the clean catch rate, so I will add that too and post both here when we have them.
Update 2 got me. Three hours of tool calls served by a weaker backup model, and the first pilot got written up as a compounding-error problem. I've been burned by the same thing, so now I log the model id that actually answered on every call, because the transcript looks identical either way. Do your pilot logs record which model served each call?
Yes, since then. Every response now carries the model that actually answered and the level that served it, and each gateway box keeps a per-request log with the same fields, so a run can be checked without trusting the transcript. It paid off again last night: our provider for the cheap tier ran out of credit mid-test, and the log showed every cheap call that night had gone to the backup model, so we knew exactly which results to rerun instead of writing up another false pattern.