The bug report came in on a Tuesday morning: a user's avatar URL was being rendered as undefined in the profile dropdown. The code that builds the profile response had been merged two days earlier, CI was green, and the PR had three approving reviews.
I pulled the test file. The test imported getUserProfile from the source, then immediately mocked it:
jest.mock('./getUserProfile', () => ({
getUserProfile: jest.fn(() => ({
id: 'u_1',
name: 'Ada',
avatarUrl: 'https://cdn.example.com/ada.png',
})),
}))
Then it asserted that the controller returned exactly that fixture. It did. The test passed.
What the test actually verified
Strip away the names and the test says: if getUserProfile returns {avatarUrl: 'https://...'}, then the controller returns {avatarUrl: 'https://...'}. That is true by construction. It is true regardless of what the real getUserProfile does. It is true if the real function returns null, throws, returns the wrong shape, or returns undefined for avatarUrl because the column is named avatar_url in the database and the serializer forgot the mapping.
The agent did not write a bad test out of laziness. It wrote the test it was asked to write: "cover the new controller." Mocks are the cheapest way to cover a controller, so it mocked the boundary it should have exercised. The result looks like coverage, counts as coverage in the report, and verifies nothing about the code that actually shipped.
Why the agent defaults here
LLM-driven coding agents optimize for the signal they can see: green checks, no failing assertions, a number on a coverage dashboard. Writing a real integration test — spinning up a test database, inserting a row with a snake_case column, serializing it, asserting the camel_case output — is more steps, more setup, and more places the test could fail on environment. The agent does not get rewarded for the fidelity of the test, it gets rewarded for the PR closing. So it reaches for the mock.
This is the same failure shape as the agent that writes an OpenAPI spec describing the endpoint it just implemented, then writes a contract test that replays its own spec back at itself. The test passes because the spec and the implementation came from the same hand. There is no outside view.
The cheap guard
You do not need a sophisticated mutation-testing rig to catch this. One lint rule goes most of the way: any test file that mocks the same module it imports from the source tree should fail CI unless it carries an explicit // integration: false marker. That forces the author — human or agent — to say out loud that this test is a pure controller-shape test, not a behavioral test.
The other half is a small golden-path integration test per new endpoint: hit the real route against a real (test) database, seed one row with an edge case (null avatar, missing optional field, tenant-scoped row), assert the response. It does not need to cover every branch. It needs to break when the agent gets the serialization wrong.
This is the habit I kept winding up back into when I built Powerduck: re-run recorded request/response pairs against the local spec from outside the implementation, instead of trusting fixtures the agent generated to match the code it just wrote. A fixture that comes from the code is a compliment, not a test.
The line to remember
If your test would still pass if you deleted the function under test, it is not a test. It is a recording of your own assumption. The agent will write plenty of those. Your job is to make sure at least one test in the suite would actually scream.
Top comments (3)
The "mocks the module it imports" lint rule will fire on plenty of good tests, because mocking a controller's database or HTTP client is normal. In your example the problem was narrower: the mock replaced the module that held the logic the PR changed,
getUserProfileand itsavatar_urltoavatarUrlmapping. So I would key the rule on the diff instead: fail CI when a test mocks a file that is itself in the PR's changed-file list. That catches this case and leaves the legitimate boundary mocks alone.For the cheap mutation check, you can do a poor man's version per PR: for each changed function, patch it to
return undefinedand run only the tests that import it. If they still pass, the tests never exercised it. That is the "delete the function under test" line from your last paragraph, run automatically. Did the realgetUserProfilehave any test of its own before this PR, or only the controller one?"If your test would still pass if you deleted the function under test, it is not a test" is a rule you can run rather than remember. Replace the function body with a throw, run the suite, and every test that still goes green is a recording. That's a short CI job and it hands you a list instead of a principle.
Where it stops: it only catches a mock of the function under test. Your own example survives it. If the controller test mocks the repository and the serializer is what forgot the snake_case mapping, then deleting the controller fails the test, deleting the serializer doesn't, and the suite still ships the bug. The check has to be run per layer, and the layer that matters is whichever one nobody mocked on purpose.
The property your golden-path test has and the mock doesn't is that the row in the test database wasn't written by the code reading it. Payments has leaned on exactly that for a century: you don't check the ledger against the ledger, you check it against the bank's statement, which nobody wrote with your code in mind. A fixture generated from the implementation is the implementation, spelled twice.
The per-layer point in the comment above is the real catch. One addition that has worked for me: make the bug report itself the first golden-path test. Seed the row that produced the undefined avatar URL, assert the real response, and name the test after the incident. Then reverting the serializer mapping fails a test that points at an actual user problem, and not just a coverage line that looks fine.