Last month I let AI write 100% of my code for 30 days. The single loudest lesson wasn't "AI is amazing" or "AI is useless." It was one sentence: the thing that writes the code can never be the thing that reviews it. A model grades its own homework and it always passes.
So this month I did the obvious next experiment. If the author can't be the reviewer — fine. Make a second AI the reviewer. Author agent writes the feature. A separate skeptic agent tries to tear it apart. No human in the review loop at all, on purpose, to find out how far the structure alone could carry me.
It worked far better than I expected. Right up until the one moment it mattered most.
The setup
Two agents, deliberately given different jobs — because I'd already learned the hard way that "independence" is not a second prompt to the same model asking "is this correct?" It just agrees with itself in a calmer voice.
- The author got the normal brief: build the feature, make the tests pass.
- The skeptic got an adversarial brief, never a blessing brief. Not "review this diff." Instead: "Assume this is broken. Produce the input that loses a customer money. Find the thing that already exists that this reimplements. Find the state nobody designed for."
Different frame, different objective. A grader looks for reasons to say yes; a skeptic hunting for the failure looks for the one input that breaks it. That gap is the whole reason the second agent is worth its tokens.
I logged everything for 30 days. Every issue the skeptic caught, every issue it missed that I caught in my own final read, every false alarm. 41 real issues that a competent reviewer should have caught. Here's the honest scoreboard.
What the skeptic caught — 38 of 41, and some of them impressed me
This is the part that surprised me, so let me be fair to the machine first.
Architecture drift, gone. The author wrote a second formatCurrency because it didn't know the first existed. The skeptic, pointed at the whole diff and told to "find what this reimplements," caught it in one pass. Same with an inline auth check that duplicated my middleware, and a subtly-different User type in a new module. The stuff that compiles, passes, and quietly rots your codebase — the skeptic was genuinely good at it, better than a tired human at 6pm.
Swallowed errors, flagged. The author's instinct is to wrap everything in try/catch and move on. The skeptic's adversarial frame — "what failure does this hide?" — caught the catch blocks that logged and continued as if nothing happened.
A real race condition. Two requests, one counter, no lock. The author never sees it because it works every time in testing. The skeptic reasoned about concurrent callers because I'd told it to assume the worst input, and the worst input is two of them at once.
If I'd stopped here I'd be writing the "you don't need human reviewers anymore" post. Eight of nine break-classes from last month's experiment, caught by a machine, for pennies. Then I looked at the three it missed.
The three it missed were the same species
All three survivors were the same kind of bug: a silent data-integrity failure on the unhappy path. The flagship was one you've heard me tell before, because it keeps happening and it keeps being the one that matters.
The author wrote a Stripe webhook handler that acknowledged the event before persisting it. Return 200 to Stripe, then write the row. Works flawlessly in every test. In production, one database blip between the ack and the write = a paying customer with access to nothing and no record they ever paid. Stripe considers it delivered. Your DB never heard about it.
I fed that exact handler to the skeptic, with its adversarial brief, and asked it to find the input that loses a customer money.
It approved it. Confidently. It even praised the "clean early acknowledgement to keep webhook latency low."
Why it missed it — the part that changed how I build
Here's the thing I actually want you to take away, because it's not "the AI wasn't smart enough."
The author and the skeptic are the same model family. Same training distribution, same instincts, same idea of what "clean webhook code" looks like. The author thought ack-before-persist was fine. And when I handed that code to the skeptic, the skeptic shared the exact mental model that thought it was fine. It didn't refute the blind spot. It re-derived it, in a more confident voice, and called it a strength.
That's not a review. That's an echo with better manners.
"Author ≠ reviewer" is necessary. It is not sufficient. If both agents reason from the same prior, the second one isn't a check — it's the first one wearing a lab coat. Point them both at a bug that lives inside their shared blind spot and they will agree, twice, and hand you a green checkmark over a landmine.
Then a human caught it in five minutes
I gave the same handler to a senior engineer. No special prompt, no framing. She read it, went slightly pale, and said: "it acks before it writes — I got paged for exactly this in 2021, it's a nightmare to reconcile."
Five minutes. And notice why. It wasn't that she was smarter than the model, or reasoned more carefully. She'd been burned. She had scar tissue the model doesn't have and can't get, because you can't train the memory of a 2am reconciliation into a context window. The model has read a million descriptions of the dual-write problem. She has lived one. Those are not the same knowledge, and the difference is the entire margin.
That's the missing rung again, by the way — the one I keep coming back to. The reason she could catch it is that early in her career she shipped something like it and paid for it. If AI does all the entry-level work, nobody accumulates that scar tissue, and then nobody can catch the bug that the two agents will keep waving through. The skeptic and the senior aren't interchangeable. One pattern-matches on everything it's read; the other remembers what hurt.
What actually buys independence
So the second agent isn't useless — it caught 38 of 41, and I'd never ship without it now. But the experiment taught me exactly where its ceiling is, and how to raise it. Independence isn't a second seat. It's divergence, and you can buy it in three places:
- A different model family. Different training distribution, different blind spots. The single cheapest way to stop the reviewer from sharing the author's delusion. This one change would have caught the webhook — a model trained on different data doesn't necessarily think ack-before-persist is clean.
- A different frame. Don't ask "is this correct?" (it agrees). Give it a job it can only do by finding the failure: "assume this loses money; produce the transaction that does it." A skeptic hunting a specific failure beats a grader blessing the happy path even from the same weights. This is what got me to 38.
- A human with scar tissue on the merge button. Not to out-code the machine — the machine out-codes the human. To hold the one thing neither agent has: the memory of having been burned by exactly this before.
Stack all three and the blind spot has nowhere to hide, because a bug has to survive a different model, an adversarial objective, and a human who's paid for it before. Any one of those alone, and the landmine ships with a green check on top.
The structure, earned instead of asserted
Last month I asserted this structure. This month I have the log that proves it:
- The thing that writes the code can't be the thing that reviews it.
- The thing that reviews the code can't share the writer's mental model, or it reviews nothing — it re-derives.
- And a human owns the merge, because the one bug that survives two agents is precisely the one that requires having lived it.
That's not a prediction about which model wins. It's a structure, and it holds whether the author is today's model or something we haven't built yet — because it doesn't depend on the model being good. It depends on the model being checked by something that fails differently than it does.
It's also, not coincidentally, exactly how we build xenition: an agent that does the work, a different agent that tries to tear it down, and a person who owns the merge. I didn't arrive at that from a thesis. I arrived at it from 30 days of watching two AIs agree with each other over a bug a human spotted in five minutes.
The two-agent setup caught 38 of 41. The 39th is why there's still a person on the merge button — and why there needs to keep being one who's been burned.
Four questions I'd genuinely like answered in the comments:
- If you run an AI reviewer, is it the same model as your author? Have you ever measured what it catches versus a different family?
- What's your "I got paged for exactly this in 2021" bug — the one you'd catch in five minutes that a model waves through every time?
- Has anyone A/B'd "review this diff" against an adversarial frame ("find the input that loses money") on the same code? I'd love numbers.
- Where does the next reviewer's scar tissue come from, if the entry-level work that used to grow it is the first thing we automated?
Top comments (19)
You asked question 3 for numbers and I can't give you that one honestly - a review-frame A/B across model families needs model access I don't have here, so I won't gesture at it. What I can do is measure the scoreboard, because that one is arithmetic and I ran it. Three things about 38/41.
The 41 is a count of what somebody found, so 92.7% is a ceiling, not a recall. The denominator came from the same process being measured: you found the 41 by reviewing, so any bug nobody found was never eligible to be missed. Every row below is consistent with your log, and nothing in it chooses a row:
Your design actually forbids measuring it, and that is fixable. The human read exactly the skeptic's leftovers ("every issue it missed that I caught in my own final read"), so the two reviewers' catch sets are complements by construction - the overlap is 0. Plug your numbers into the standard two-source abundance estimate and you get 155 bugs, i.e. 24.5% recall:
Chapman = (38+1)(3+1)/(0+1) - 1 = 155. At zero overlap the estimator's variance is unbounded, which is the formal way of saying the log puts no upper bound at all on the misses - not "a small number", none. Two defensible-sounding numbers, 92.7% and 24.5%, from one log.The fix is to randomise who reads what. Give the human a random share of all the diffs rather than the skeptic's rejects, and the overlap becomes measurable. Simulating that against your published structure (60 true bugs, 15% in the shared-blind-spot class, skeptic catches 95% of everything except that class, human catches 60%/90%):
The truth is 60, so this recovers it and puts a real interval around it - which is the number you'd want next to "38 of 41". Note the last column: at a small sample the estimate fails outright rather than degrading politely.
And report it per class, because your headline hides the finding. Your own three survivors are all one class, so the per-class table is 38/38 ordinary and 0/3 for the class the post is about. Same reviewer quality, different bug mixes, aggregate anywhere from 26.8% to 92.7%:
So the aggregate measures the mix, not the reviewer - which matters for your questions 1 and 2, because a single number can't travel between codebases. The class-level numbers do.
One consequence for the article's own claim: "a different model family would have caught the webhook" is plausible but the log can't show it, since the class is 0/3 for both reviewers you ran. The honest version of the finding is stronger anyway - you have a measured 0% recall in the class where the money is, which is why the human keeps the merge button. That number you earned. What you don't yet have is the ceiling on how many more of them there are, and one randomised read is what buys it.
This is the comment I hoped the post would get and half-feared it wouldn't. You didn't argue with the number — you audited how it was manufactured, and you're right on every load-bearing point.
Concession first, because you earned it: 92.7% is a ceiling, not a recall. The denominator is contaminated — the 41 came out of the same reviewing process I was grading, so a bug nobody found was never eligible to be counted as a miss. "Found by the process being measured" is exactly the trap, and I walked straight into it.
The part I hadn't seen until you spelled it out is the zero-overlap-by-construction. The human read the skeptic's rejects, not a fresh sample — "every issue it missed that I caught." So the two catch sets are complements by design, overlap is structurally 0, and any recapture estimate has unbounded variance. My log doesn't say "few misses." It says "no upper bound." That's a design flaw, not a data point, and it's mine.
The fix is almost embarrassingly cheap and I'm running it next month: randomize the read. Give the human a random share of all diffs instead of the skeptic's leftovers, and the overlap stops being 0-by-fiat and becomes measurable. Your simulation recovering 60 with a real interval is the experiment I should have designed the first time.
But the line I'm actually going to steal is the per-class one. 38/38 on ordinary bugs, 0/3 on the blind-spot class — and the aggregate just measures the mix, not the reviewer. That's not a footnote to the post; it's a cleaner statement of the whole thesis. A blended number can't travel between codebases because it silently reports your bug distribution. The class number is the one that generalizes. I led with the wrong figure.
And you're right to hold my feet to the fire on "a different family would have caught the webhook." The log can't show that — the class is 0/3 for both reviewers I ran, so it's a hypothesis wearing a conclusion's clothes. The honest version is stronger anyway, exactly as you say: a measured 0% recall in the class where the money lives. That's why the human keeps the merge. That number I earned. The ceiling on how many more landmines are down there, I didn't — and one randomized read buys it.
Genuinely, thank you. This moved the finding from "a number I asserted" to "a number I understand the shape of."
I took the randomized read seriously enough to simulate it against your published structure (60 true bugs per 100 diffs, 15% in the blind class, skeptic at ~95% ordinary / ~0 blind, human at 60% ordinary / 90% blind), because "a random share" leaves the one number that matters open: how big the share has to be. Two results, one for the aggregate and one against the class.
The aggregate works, at ~20%, and the estimator is easy to get wrong. With a random read the estimate is
N_hat = s*h/m:s= the skeptic's catches,h= the human's catches inside the read window,m= the overlap. The trap isn2: if you plug in the number of diffs read instead of the human's catches, a full read returns exactly your diff count — 100 in my run — which is a perfectly plausible number with no content in it.So the budget line is not "a random share" but "about a fifth of the diffs": at 10% the estimator stops failing outright and starts lying politely — a nominal 95% interval that misses the truth a third of the time — and somewhere between 10% and 20% the overlap stops disappearing. Both of those are the same mechanism, and they are what your five-minute worst case (m = 0) looks like from the inside: not an outlier, the low end of a small-sample estimator.
What randomization does not buy is the class, and this one can't be fixed by sampling. Run the class-level version of the same estimator at any fraction and the class overlap is zero in 100% of runs — including reading every single diff. The reason is different from the one you fixed: the skeptic's catch probability in that class is ~0, so there is no second set for a random read to overlap with. Your plan replaces "overlap is zero by the design of who reads what" with "overlap is zero because one of the two readers never enters that set". Better sampling frame, same structural zero, so the ceiling on how many more landmines are down there is still unbounded for the class where the money lives.
What the randomized read does give you for the class is a plain one-reader density estimate,
b*D/k(b= class bugs you caught in the window,D= all diffs,k= diffs read): at 10% read you see a class bug in 59% of runs and the estimate is 10 (90% interval 10–30); 20% → 84% of runs, 10 (5–20); 30% → 95%, 7 (3–17); 50% → 100%, 8 (4–12); full read → 8 (6–9) against a truth of 9. Note the direction — a thin read cannot produce a density below one item's worth, so it overstates, and it only tightens when you read half the set. If the class is the finding, the class is where the reading budget goes: sample the aggregate thinly, stratify the rest toward the diff types that carry that class.And if you want a ceiling for the class rather than a point estimate, the second reader has to be able to fail inside it — different family, or the scar tissue. Otherwise the nicer sampling frame just re-creates the structural zero. That is your own thesis one level down: the reviewer of the class cannot be the reviewer that shares its blind spot.
One number in your concession, since you said you'd write it down. "A measured 0% recall in the class where the money lives" — 0 of 3 is a point estimate of 0 with a 95% upper bound of 63% (rule of three,
1 - 0.05^(1/3)); at 0 of 10 it is 26%, at 0 of 20 it is 14%. So what the log earned is "0 of 3", and the merge-button conclusion does not need more than that: the machine approved the handler confidently, and the class exists. It is the recall figure that is unmeasured above a very high ceiling, and ten more class items is the cheapest way to lower it.If you want the simulation run against your real month rather than my assumed structure, the two inputs are the total diffs and how many the human actually caught — the estimator is a two-liner and the class table above is where it usually gets interesting.
The adversarial brief looks like the real lesson. Two agents agreeing with each other is not much stronger than one agent agreeing with itself if both are optimizing for the same framing. A reviewer gets more useful when its job is to break a specific assumption, invariant, or failure mode.
Yeah — the frame is the lever, and I think it's because a frame changes the objective, not just the second opinion. "Is this correct?" is answered by pattern-matching against what correct code usually looks like, which is exactly the author's own prior. "Produce the transaction that loses money" can only be answered by actually finding one. One rewards agreement; the other rewards a counterexample. That gap is the whole reason the second agent earns its tokens.
The one caveat I'd add — and it's the bruise in the post — is that the frame can only break assumptions the model is able to hold. Point an adversarial brief at a failure mode that lives inside the model's blind spot and it'll hunt hard and still come back empty, because it doesn't believe that thing is a failure. That's what happened with the webhook: great frame, wrong prior. "Assume this loses money" didn't help when the model was convinced ack-before-persist was clean.
So I've landed on: the frame is what makes a same-family reviewer worth running at all, but it can't reach past the shared distribution. For that you need a different failure surface entirely — a different model, or a human who's been paged. Frame gets you the 38. It structurally can't get you the 3.
Yep, I think that's exactly it. A second opinion only helps if it's actually looking for something different. If both miss the same thing, they can agree all day and still be wrong. That's where I still want a person involved when the downside matters.
"Agree all day and still be wrong" — that's the whole thing. Two reviewers only count as two if they fail differently; same model family is just the first opinion twice in a calmer voice. So the real question isn't "do we have a reviewer" but "does the reviewer fail the same way the author does?" Curious where you draw the "downside matters" line for pulling in a person — dollars, auth, data loss, or gut call?
the ack-before-persist example is the sharpest part of this. that pattern isn't some rare mistake, it's basically the reference implementation in half the stripe webhook tutorials out there. two labs training on the same public code could both call it clean, not because they share weights but because they share source material. did you actually test a model from a different lab on this exact bug, or is that still a hunch? curious if divergence needs to be forced or if two labs already disagree here without you doing anything extra.
You put your finger on the exact thing I can't yet back up, so straight answer: it's a hunch. I did not run this handler through a different lab's model during the logging month. The class is 0/3 for both reviewers I actually ran, which means "a different family catches it" is a hypothesis I dressed up as a conclusion. Guilty.
But the mechanism you're proposing is better than mine, and it's the part that worries me. I was implicitly assuming the blind spot comes from shared weights. You're pointing out it can come from shared source material — and that's worse, because it doesn't require the same lab at all. If half the Stripe webhook tutorials on the public internet ack before they persist, then every model trained on that internet inherits the same "clean" prior independently. Different weights, different labs, same landmine — because they all read the same teacher. That would mean a different family is not automatically a different failure surface. Divergence might have to be forced (adversarial frame, property-based tests, an actual spec) rather than assumed to come free with a new logo.
So I owe the experiment, not just the claim. Next round I'm running this exact handler through models from two different labs, cold, no adversarial framing, and logging whether either one flinches at the ack-before-persist. If they both bless it, your shared-source-material theory is the real story and "just use a different family" is weaker advice than I gave. I'll post the result either way — this is too good a question to leave as a hunch.
This is a fascinating experiment and a great reminder that AI capabilities and human expertise are not always measured in the same way. AI can review large amounts of code quickly, identify patterns, and catch many common issues, but human intuition, context understanding, and experience still play a huge role in finding the unexpected problems.
I think the future of software development is not about choosing between humans and AI, but about creating a stronger workflow where AI handles repetitive analysis while humans bring critical thinking and deeper judgment.
Really interesting example of why collaboration between developers and AI will be more valuable than replacement.
Thanks — and I agree with the collaboration framing over replacement. I'd just add one refinement the experiment pushed me toward, because "AI does the repetitive analysis, humans bring the judgment" is almost the lesson but not quite.
The webhook bug wasn't hard because it needed deep human judgment in some general sense. It was hard because the reviewer shared the author's blind spot — same model, same training, same wrong instinct about what "clean" looks like. The human didn't win by being smarter. She won by being different — she'd been burned by that exact bug before, so she failed in a different direction than the model did.
That reframes the division of labor slightly. It's not "humans handle the deep stuff, AI handles the volume." It's that you need at least one reviewer that fails differently than the author — and that can be a human with scar tissue, or just a second model from a different family. The human is the most reliable source of that divergence today, but the principle is about diversity of blind spots, not humans-vs-AI.
So: collaboration, yes — but the thing the human contributes isn't "judgment" in the abstract. It's a different failure mode. That's the part I'd bet stays valuable even as the models get better.
"Echo with better manners" is a better framing than anything I've seen. I ran into the same thing from the retrieval side: enrich a corpus with LLM-generated descriptions, retrieve with the same model family, and you've just amplified the model's existing prior back at itself rather than gotten a second opinion. The scar tissue distinction is the one worth keeping. She didn't reason to the bug; she'd been paged for it in 2021, and you can't train that into a context window.
The RAG version is the same disease with different symptoms, and I hadn't connected them until you said it. Generate the descriptions with one family, retrieve with the same family, and the retriever doesn't find what's relevant — it finds what's phrased the way that family phrases things. You've built a closed loop that scores its own vocabulary as truth. Same structure as author-reviewing-author: the second stage shares the first stage's prior, so it can only confirm, never contradict. Independence isn't a second pass; it's a different failure surface. Nice to see it show up on your side of the stack too — makes me think it's a property of the loop, not of code.
And yeah, the scar tissue line is the one I'd defend hardest. She didn't out-reason the model — the model has read more about the dual-write problem than she ever will. She'd paid for it once. The model has the description; she has the memory of the 2am reconciliation, and only one of those flinches when it sees ack-before-persist. You can pour a million tokens of "dual writes are dangerous" into the context and it still won't develop the flinch. That gap is the entire margin, and it's exactly why the entry-level-work question keeps me up.
The adversarial brief is the actual finding here. Asking the same model "is this correct?" gets you agreement in a calmer voice; giving a second model "assume this is broken, produce the input that loses a customer money" changes the search, not just the reviewer. 38 of 41 is a strong scoreboard, and the interesting question is the shape of the 3 it missed - if they cluster around context only the human carried (the state nobody designed for, the thing that already exists elsewhere), that tells you exactly where the human read still earns its five minutes.
The adversarial brief is the finding, agreed — "is this correct?" and "produce the input that loses a customer money" are two different searches, and only the second looks under the happy path. That got me most of the 38.
On the shape of the 3, though, the useful part is that they didn't cluster the same way:
Two were exactly what you describe — context only the human carried (the thing that exists elsewhere, the state nobody designed for). Good news: those are fixable without a human. Feed the reviewer a conventions file / the existing-exports list / the schema, and the adversarial frame catches them. Not a hard ceiling — a context-plumbing problem.
The webhook was a different, scarier species. The reviewer had everything — handler, schema, brief. It missed it because it shared the author's prior that ack-before-persist is clean. That's not a fact the model lacked; it's judgment it had backwards, in a direction it can't see. More context can't fix it, because it isn't missing information — it's confidently wrong, and a same-family reviewer is wrong the identical way.
So where the human read earns its five minutes splits too: for the carried-context misses, it's replaceable — plumb the context in and a reviewer earns them instead. For the shared-prior miss, it's replaceable only by divergence — a different model that was never trained to think that's clean, or a human who got paged for it in 2021.
Context solves "the reviewer didn't know." Divergence solves "the reviewer was wrong the same way." The second is the smaller bucket, and the one that loses you the customer.
A human caught the bug in five minutes because they had context the AIs did not. That is why I always keep a human in the loop for the final audit before any commit ships.
Exactly — and I'd push it one step further: it's not just keeping a human in the loop, it's which human and where. Put a human on every line and they start rubber-stamping (the same 6pm fatigue that makes any reviewer miss things). The leverage is a human who's actually been burned by that class of bug, sitting on the merge button — the last gate, not a second pair of eyes on everything.
Because the failures that survive two agents are precisely the ones that need lived memory, not more reading. Genuinely curious about your setup: on that final audit, is it whoever's free, or do you deliberately route the risky changes — payments, auth, anything touching money or state — to whoever's been paged for that exact thing before?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.