2 is the one that's bitten the most people I know: green tests that only pass because the assertion quietly changed. A candidate for #8: silent scope creep, where you ask for a one-line fix and the diff also "tidies" three unrelated files. The 30-second check is git diff --stat before reading a single line. If it touched files you didn't mention, ask why before reviewing anything else. Great list, and the version-mismatch one deserves way more attention than it gets.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you, that's a good eighth. A one-line fix that quietly touches three other files is easy to miss, because each extra change looks harmless on its own. Checking git diff --stat first is a cheap way to see the real size of the change before reading any line. Do you see it more when the agent runs with auto-accept on, or even when you review each step?
Auto-accept makes it far more likely, since nobody sees the intermediate steps. But step-by-step review doesn't fully catch it either, because each individual edit looks reasonable in isolation, like a renamed variable here or a "cleaner" import there. That's why the --stat check at the end matters: it's the only view that shows the total footprint of the change at once.
The test-diff check (#2) is the one I'd put first — it catches the sneaky failure mode. My addition: also grep the diff for assertion weakening, not just skips. The tell-tale pattern is toBe/toEqual downgrading to toBeTruthy/toBeDefined, or a timeout extended "to fix flakiness" — usually means the test now passes by construction.
4's "which 3 inputs would break this?" is genuinely better than asking for edge cases in the abstract. Curious whether you've seen the swallowed-catch one (#5) show up more in AI-generated Express handlers than elsewhere — that's where I keep finding catch { return [] }.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you, glad the test-diff one was useful! You're right about weakened assertions. proof-of-done already flags toBe/toEqual turning into toBeTruthy/toBeDefined, here's a short demo and the code. The longer timeout "to fix flakiness" is a good point, it doesn't check that yet. Have you seen it more in Jest unit tests or in e2e tests like Playwright?
Founder of Hex16, a small web studio in Poland. I build Next.js sites and headless Shopify stores, with GSAP, Three.js and shaders where they earn their place, and look after technical SEO.
Mine is a cousin of #3. The package exists and is popular, it just pulls in a pile of dependencies for something that was fifteen lines of code. I accepted one from an agent without looking. The 30-second check: git diff --stat on the lockfile. If it grew by hundreds of lines for a small feature, ask why.
An eighth, from yesterday: the fix passes everything, and it never ships.
The AI fixed a bug with a new patch folder, and every test passed on my machine. But the deploy script packs a fixed list of folders, and the new one wasn't on it. No test could see that, because tests run where the file exists.
A 30-second check that could help: list the new files, then look for each one in what you ship.
git diff --name-status main | grep '^A'
Compare that with your Dockerfile COPY lines, or your package's "files" list. I only caught mine by reading the script.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you for sharing this one. The tests pass on your machine because the file is right there, and the package just never gets it. I'll think about adding a check like that to proof-of-done. Out of curiosity, is your folder list hardcoded in a shell script, or does it live in a config file? That would change how a check could read it.
Hardcoded on purpose, in a shell script: a fixed list keeps secrets like .env from shipping by accident.
We'd check the package, not the script, so it works either way. And our team always likes a human in the loop here: the check flags the file, and a human decides if it ships.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you for explaining. An allowlist is a sensible choice here: it fails safe, because anything not on the list stays out, including a stray .env. Checking the built package also keeps the check independent of how the list is maintained, whether it's a shell script or a config file. And I agree that it should flag rather than block: a new file is not always a mistake, so the final call belongs to a person.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The eighth one from baigorria is the one I'd add — a fix that passes everything and never ships, because the deploy script packs a fixed folder list and the new folder wasn't on it. No test could see it, because tests run where the file exists. That's the gap between test env and ship env, and it's the check I'd steal first: does the deploy actually include what changed?
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you Paul, always nice to see you here! I agree, the gap between where tests run and what actually ships is an easy one to miss. A test can only check what exists where it runs.
The line that the worst AI code is the kind that looks right is accurate. Crashers get noticed. Almost-right patches get merged.
The seven checks are concrete enough to use as a habit. Asking which command ran and how many tests actually executed catches the empty green. Diffing test files catches the weakened assertion. Asking which three inputs would break this forces the happy-path case into the open. Searching the diff for catch blocks that return [] turns a silent support ticket into a review item.
When teams ask Ascendra Ventures to take an AI-built prototype into production, those checks become part of the written acceptance bar. The estimate starts after the bar is clear, not before.
Which of the seven do you catch most often in practice?
This is such a sharp, practical breakdown of the "hallucination of correctness."
Number 2 (test weakening) and Number 5 (error swallowing) are definitely the most insidious. There is nothing worse than hunting down a production bug only to realize the AI "fixed" a test by swapping a strict assertion for .toBeDefined() or wrapping a network failure in a silent catch { return []; }.
My candidate for the 8th way: The State Drift / Stale Dependency Trap. The AI writes a custom Hook or asynchronous routine that looks completely clean and passes the initial render test, but quietly misses a reactive dependency or creates a closure over stale state. It works fine during a quick manual check, but drifts out of sync the second real users start interacting with the UI sequentially.
The git diff --stat habit is pure gold. Simple, fast, and eliminates half the noise before you even start reviewing lines. Great write-up!
The '0 tests passed is not a pass' check is the one we hit hardest in Electron work: an AI-generated IPC handler looks correct, but it never validates the renderer payload, and the integration test that should catch that is mocked away. The habit of asking 'which command ran and what was the exit code' generalizes well to desktop — especially around auto-update and crash-reporting code, where the happy path is easy to generate and the failure path (network down, partial download, stale dump) is exactly what gets skipped. Pairing each AI change with a forced failure-path test has saved us more than the happy-path tests did.
AI can save us a lot of time, but blindly trusting its output is where things get tricky. Always review, test, and understand the code before shipping it. And if possible, use AI-powered code review tools to catch potential issues and automate parts of the review process. Never let AI be the only one approving its own work.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · Claude Code · Codex · Building tools to make your daily dev work easier
AI makes coding faster, no doubt! But it's always a good idea to double-check what it generates. Use code review tools or AI reviewers to catch things you might miss. A little extra checking can save a lot of trouble later.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · Claude Code · Codex · Building tools to make your daily dev work easier
2 is the one that's bitten the most people I know: green tests that only pass because the assertion quietly changed. A candidate for #8: silent scope creep, where you ask for a one-line fix and the diff also "tidies" three unrelated files. The 30-second check is git diff --stat before reading a single line. If it touched files you didn't mention, ask why before reviewing anything else. Great list, and the version-mismatch one deserves way more attention than it gets.
Thank you, that's a good eighth. A one-line fix that quietly touches three other files is easy to miss, because each extra change looks harmless on its own. Checking
git diff --statfirst is a cheap way to see the real size of the change before reading any line. Do you see it more when the agent runs with auto-accept on, or even when you review each step?Auto-accept makes it far more likely, since nobody sees the intermediate steps. But step-by-step review doesn't fully catch it either, because each individual edit looks reasonable in isolation, like a renamed variable here or a "cleaner" import there. That's why the --stat check at the end matters: it's the only view that shows the total footprint of the change at once.
The test-diff check (#2) is the one I'd put first — it catches the sneaky failure mode. My addition: also grep the diff for assertion weakening, not just skips. The tell-tale pattern is
toBe/toEqualdowngrading totoBeTruthy/toBeDefined, or a timeout extended "to fix flakiness" — usually means the test now passes by construction.4's "which 3 inputs would break this?" is genuinely better than asking for edge cases in the abstract. Curious whether you've seen the swallowed-catch one (#5) show up more in AI-generated Express handlers than elsewhere — that's where I keep finding
catch { return [] }.Thank you, glad the test-diff one was useful! You're right about weakened assertions. proof-of-done already flags
toBe/toEqualturning intotoBeTruthy/toBeDefined, here's a short demo and the code. The longer timeout "to fix flakiness" is a good point, it doesn't check that yet. Have you seen it more in Jest unit tests or in e2e tests like Playwright?Mine is a cousin of #3. The package exists and is popular, it just pulls in a pile of dependencies for something that was fifteen lines of code. I accepted one from an agent without looking. The 30-second check: git diff --stat on the lockfile. If it grew by hundreds of lines for a small feature, ask why.
An eighth, from yesterday: the fix passes everything, and it never ships.
The AI fixed a bug with a new patch folder, and every test passed on my machine. But the deploy script packs a fixed list of folders, and the new one wasn't on it. No test could see that, because tests run where the file exists.
A 30-second check that could help: list the new files, then look for each one in what you ship.
git diff --name-status main | grep '^A'
Compare that with your Dockerfile COPY lines, or your package's "files" list. I only caught mine by reading the script.
Thank you for sharing this one. The tests pass on your machine because the file is right there, and the package just never gets it. I'll think about adding a check like that to proof-of-done. Out of curiosity, is your folder list hardcoded in a shell script, or does it live in a config file? That would change how a check could read it.
Hardcoded on purpose, in a shell script: a fixed list keeps secrets like .env from shipping by accident.
We'd check the package, not the script, so it works either way. And our team always likes a human in the loop here: the check flags the file, and a human decides if it ships.
Thank you for explaining. An allowlist is a sensible choice here: it fails safe, because anything not on the list stays out, including a stray .env. Checking the built package also keeps the check independent of how the list is maintained, whether it's a shell script or a config file. And I agree that it should flag rather than block: a new file is not always a mistake, so the final call belongs to a person.
The eighth one from baigorria is the one I'd add — a fix that passes everything and never ships, because the deploy script packs a fixed folder list and the new folder wasn't on it. No test could see it, because tests run where the file exists. That's the gap between test env and ship env, and it's the check I'd steal first: does the deploy actually include what changed?
Thank you Paul, always nice to see you here! I agree, the gap between where tests run and what actually ships is an easy one to miss. A test can only check what exists where it runs.
The line that the worst AI code is the kind that looks right is accurate. Crashers get noticed. Almost-right patches get merged.
The seven checks are concrete enough to use as a habit. Asking which command ran and how many tests actually executed catches the empty green. Diffing test files catches the weakened assertion. Asking which three inputs would break this forces the happy-path case into the open. Searching the diff for catch blocks that return [] turns a silent support ticket into a review item.
When teams ask Ascendra Ventures to take an AI-built prototype into production, those checks become part of the written acceptance bar. The estimate starts after the bar is clear, not before.
Which of the seven do you catch most often in practice?
This is such a sharp, practical breakdown of the "hallucination of correctness."
Number 2 (test weakening) and Number 5 (error swallowing) are definitely the most insidious. There is nothing worse than hunting down a production bug only to realize the AI "fixed" a test by swapping a strict assertion for
.toBeDefined()or wrapping a network failure in a silentcatch { return []; }.My candidate for the 8th way: The State Drift / Stale Dependency Trap. The AI writes a custom Hook or asynchronous routine that looks completely clean and passes the initial render test, but quietly misses a reactive dependency or creates a closure over stale state. It works fine during a quick manual check, but drifts out of sync the second real users start interacting with the UI sequentially.
The
git diff --stathabit is pure gold. Simple, fast, and eliminates half the noise before you even start reviewing lines. Great write-up!The '0 tests passed is not a pass' check is the one we hit hardest in Electron work: an AI-generated IPC handler looks correct, but it never validates the renderer payload, and the integration test that should catch that is mocked away. The habit of asking 'which command ran and what was the exit code' generalizes well to desktop — especially around auto-update and crash-reporting code, where the happy path is easy to generate and the failure path (network down, partial download, stale dump) is exactly what gets skipped. Pairing each AI change with a forced failure-path test has saved us more than the happy-path tests did.
AI can save us a lot of time, but blindly trusting its output is where things get tricky. Always review, test, and understand the code before shipping it. And if possible, use AI-powered code review tools to catch potential issues and automate parts of the review process. Never let AI be the only one approving its own work.
Thank you for reading! Fully agree, the AI should never be the only one approving its own work. Which review tool do you use?
AI makes coding faster, no doubt! But it's always a good idea to double-check what it generates. Use code review tools or AI reviewers to catch things you might miss. A little extra checking can save a lot of trouble later.
Thank you! Agreed, a few minutes of checking is much cheaper than a bug in production.