The worst AI code isn't the code that crashes. Code that crashes, you notice.
The worst one looks right, passes a quick glance, and the AI says "Done! ✅". Then it breaks in production two weeks later.
You're not alone in this. In Stack Overflow's 2025 developer survey, the most common complaint about AI tools was "AI solutions that are almost right, but not quite" (66% of developers). In the 2026 survey, nearly half said they trust AI output only when they can check it easily.
So here's my list of how "almost right" sneaks in, with a check for each that takes about 30 seconds. I use AI coding tools every day, and I keep running into every one of these.
1. "Fixed!" … but nothing was run
The AI edits the code, explains the fix nicely and stops. No test ran. Sometimes a test command did run, but it found 0 tests and still exited green.
30-second check: ask "Which command did you run, and what was the exit code?" Then look at the output for the number of tests that actually ran. 0 passed is not a pass.
2. The tests pass because the test changed
The fastest way to make a red test green is to change the test, not the code. Look out for:
-
it.skip(/xit(/@pytest.mark.skip -
expect(total).toBe(90)quietly becomingexpect(total).toBeDefined() - a snapshot "updated" to match the new (wrong) output
30-second check: git diff -- '*test*' '*spec*' before you accept anything. If a test file changed and you didn't ask for that, ask why.
3. A function or package that doesn't exist
It reads perfectly: import { parseSafe } from "zod/utils". That function is not there. Or it suggests npm install of a package with a very believable name. Attackers now register those made-up names on purpose, so that slip can turn into a real risk.
30-second check: open the real docs or the npm page. A brand-new package with a handful of downloads is a red flag.
4. Only the happy path works
The demo input works. An empty list, null, a timezone, a 0, a very long string or a double click doesn't.
30-second check: ask the AI, "Which 3 inputs would break this?" Then run those 3. It is surprisingly honest when you ask that directly.
5. Errors get swallowed
The code stops crashing because the crash is hidden:
try {
return await getOrders(userId);
} catch {
return []; // the user just sees "no orders"
}
Nothing fails, nothing logs, and the bug becomes a support ticket.
30-second check: search the diff for catch, || [], ?? null and except: pass. Each one should log, rethrow or have a reason.
6. The right answer for the wrong version
The code is correct for a version you don't use: the Next.js pages router in an app-router project, an old React Router API, a removed config option. It compiles in the AI's head, not in your project.
30-second check: compare the imports with the versions in your package.json (or lock file), and paste your version into the prompt next time.
7. Security that "works"
The login works, so it looks done. But the JWT is decoded instead of verified, the secret is hard-coded, or an API route never checks who is calling it.
30-second check: for every new route, ask "Who is allowed to call this, and where is that checked?" Search the diff for strings that look like keys.
The habit that catches most of these
Before accepting any AI change, run one command:
git diff --stat
Did it touch files you never mentioned? Did test files change? Is the diff 10× bigger than the task? That one look catches half of this list.
Your turn 👇
Which one has bitten you most recently? And what's the 8th way that's missing from this list? I'll add the best ones (with credit) to a follow-up post.
Small note: I got tired of doing checks 1 and 2 by hand, so I built a free Claude Code plugin, proof-of-done, that does them automatically, and a free security-audit skill for number 7.
Happy Diwali in advance 🪔: may your builds be green and your tests honest.
Everything I've built (all free and open source)
which-ai-to-use: picks the best AI and model for each task, and hands your work to another AI when you hit a usage limit
- Try it:
npx which-ai-to-use - npm · VS Code Marketplace · Open VSX (Cursor, Windsurf) · GitHub · Website
proof-of-done: a Claude Code plugin that won't let the AI say "done" until the tests pass
- Install:
/plugin marketplace add WST2002/claude-code-skillsthen/plugin install proof-of-done@claude-code-skills - GitHub · Featured on the HOL blog
security-audit: a Claude Code skill that finds security holes before attackers do
All my Claude Code skills: github.com/WST2002/claude-code-skills
Watch the demos on YouTube: @wst2002rk · proof-of-done Short · which-ai-to-use Short · security-audit Short
Read more: Switch AIs mid-task without losing context · Your AI said "fixed" but never ran the tests · My AI finds security holes before attackers do
Top comments (22)
2 is the one that's bitten the most people I know: green tests that only pass because the assertion quietly changed. A candidate for #8: silent scope creep, where you ask for a one-line fix and the diff also "tidies" three unrelated files. The 30-second check is git diff --stat before reading a single line. If it touched files you didn't mention, ask why before reviewing anything else. Great list, and the version-mismatch one deserves way more attention than it gets.
Thank you, that's a good eighth. A one-line fix that quietly touches three other files is easy to miss, because each extra change looks harmless on its own. Checking
git diff --statfirst is a cheap way to see the real size of the change before reading any line. Do you see it more when the agent runs with auto-accept on, or even when you review each step?Auto-accept makes it far more likely, since nobody sees the intermediate steps. But step-by-step review doesn't fully catch it either, because each individual edit looks reasonable in isolation, like a renamed variable here or a "cleaner" import there. That's why the --stat check at the end matters: it's the only view that shows the total footprint of the change at once.
The test-diff check (#2) is the one I'd put first — it catches the sneaky failure mode. My addition: also grep the diff for assertion weakening, not just skips. The tell-tale pattern is
toBe/toEqualdowngrading totoBeTruthy/toBeDefined, or a timeout extended "to fix flakiness" — usually means the test now passes by construction.4's "which 3 inputs would break this?" is genuinely better than asking for edge cases in the abstract. Curious whether you've seen the swallowed-catch one (#5) show up more in AI-generated Express handlers than elsewhere — that's where I keep finding
catch { return [] }.Thank you, glad the test-diff one was useful! You're right about weakened assertions. proof-of-done already flags
toBe/toEqualturning intotoBeTruthy/toBeDefined, here's a short demo and the code. The longer timeout "to fix flakiness" is a good point, it doesn't check that yet. Have you seen it more in Jest unit tests or in e2e tests like Playwright?Mine is a cousin of #3. The package exists and is popular, it just pulls in a pile of dependencies for something that was fifteen lines of code. I accepted one from an agent without looking. The 30-second check: git diff --stat on the lockfile. If it grew by hundreds of lines for a small feature, ask why.
An eighth, from yesterday: the fix passes everything, and it never ships.
The AI fixed a bug with a new patch folder, and every test passed on my machine. But the deploy script packs a fixed list of folders, and the new one wasn't on it. No test could see that, because tests run where the file exists.
A 30-second check that could help: list the new files, then look for each one in what you ship.
git diff --name-status main | grep '^A'
Compare that with your Dockerfile COPY lines, or your package's "files" list. I only caught mine by reading the script.
Thank you for sharing this one. The tests pass on your machine because the file is right there, and the package just never gets it. I'll think about adding a check like that to proof-of-done. Out of curiosity, is your folder list hardcoded in a shell script, or does it live in a config file? That would change how a check could read it.
Hardcoded on purpose, in a shell script: a fixed list keeps secrets like .env from shipping by accident.
We'd check the package, not the script, so it works either way. And our team always likes a human in the loop here: the check flags the file, and a human decides if it ships.
Thank you for explaining. An allowlist is a sensible choice here: it fails safe, because anything not on the list stays out, including a stray .env. Checking the built package also keeps the check independent of how the list is maintained, whether it's a shell script or a config file. And I agree that it should flag rather than block: a new file is not always a mistake, so the final call belongs to a person.
The eighth one from baigorria is the one I'd add — a fix that passes everything and never ships, because the deploy script packs a fixed folder list and the new folder wasn't on it. No test could see it, because tests run where the file exists. That's the gap between test env and ship env, and it's the check I'd steal first: does the deploy actually include what changed?
Thank you Paul, always nice to see you here! I agree, the gap between where tests run and what actually ships is an easy one to miss. A test can only check what exists where it runs.
The line that the worst AI code is the kind that looks right is accurate. Crashers get noticed. Almost-right patches get merged.
The seven checks are concrete enough to use as a habit. Asking which command ran and how many tests actually executed catches the empty green. Diffing test files catches the weakened assertion. Asking which three inputs would break this forces the happy-path case into the open. Searching the diff for catch blocks that return [] turns a silent support ticket into a review item.
When teams ask Ascendra Ventures to take an AI-built prototype into production, those checks become part of the written acceptance bar. The estimate starts after the bar is clear, not before.
Which of the seven do you catch most often in practice?
This is such a sharp, practical breakdown of the "hallucination of correctness."
Number 2 (test weakening) and Number 5 (error swallowing) are definitely the most insidious. There is nothing worse than hunting down a production bug only to realize the AI "fixed" a test by swapping a strict assertion for
.toBeDefined()or wrapping a network failure in a silentcatch { return []; }.My candidate for the 8th way: The State Drift / Stale Dependency Trap. The AI writes a custom Hook or asynchronous routine that looks completely clean and passes the initial render test, but quietly misses a reactive dependency or creates a closure over stale state. It works fine during a quick manual check, but drifts out of sync the second real users start interacting with the UI sequentially.
The
git diff --stathabit is pure gold. Simple, fast, and eliminates half the noise before you even start reviewing lines. Great write-up!The '0 tests passed is not a pass' check is the one we hit hardest in Electron work: an AI-generated IPC handler looks correct, but it never validates the renderer payload, and the integration test that should catch that is mocked away. The habit of asking 'which command ran and what was the exit code' generalizes well to desktop — especially around auto-update and crash-reporting code, where the happy path is easy to generate and the failure path (network down, partial download, stale dump) is exactly what gets skipped. Pairing each AI change with a forced failure-path test has saved us more than the happy-path tests did.
AI can save us a lot of time, but blindly trusting its output is where things get tricky. Always review, test, and understand the code before shipping it. And if possible, use AI-powered code review tools to catch potential issues and automate parts of the review process. Never let AI be the only one approving its own work.
Thank you for reading! Fully agree, the AI should never be the only one approving its own work. Which review tool do you use?
AI makes coding faster, no doubt! But it's always a good idea to double-check what it generates. Use code review tools or AI reviewers to catch things you might miss. A little extra checking can save a lot of trouble later.
Thank you! Agreed, a few minutes of checking is much cheaper than a bug in production.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.