“Fixed the bug. All tests pass.”
Great.
Which tests? Did they run? Did they cover the bug—or did the agent change the test until it passed?
A confident completion message is easy to read as proof. But before you merge the changes, you need to understand what actually happened.
Here’s a simple workflow for checking an AI agent’s work without manually repeating every step.
1. Define “done” before it starts
“Fix the checkout bug” leaves too much open to interpretation.
Give the agent an observable outcome:
When a discount exceeds the order subtotal, the total should be zero. Add a regression test. Preserve valid discount behavior.
Now you have something concrete to verify.
For a UI task, describe the interaction. For an API task, specify the response and side effects.
Clear acceptance criteria make both implementation and verification easier.
2. Ask for evidence in the completion report
A useful report should include:
- What changed.
- Which commands ran.
- Whether they passed or failed.
- What could not be checked.
Compare these two reports:
“Everything works.”
And:
“Added a regression test for excessive discounts. The checkout test suite passed. Browser checkout was not tested because payment credentials were unavailable.”
The second report tells you what you can rely on and what still needs attention.
When your tool exposes execution logs, inspect them. A command mentioned in a summary is still a claim until you can confirm its execution.
3. Make sure the test catches the original bug
A passing test is useful only if it checks the behavior you care about.
Suppose the task is to prevent negative totals.
This assertion tells you almost nothing:
expect(result).toBeDefined();
This one checks the requirement:
expect(calculateTotal(50, 80)).toBe(0);
Where practical, confirm that the regression test fails against the original implementation and passes after the fix.
Then check nearby behavior:
expect(calculateTotal(50, 20)).toBe(30);
The goal is to demonstrate that the bug was fixed while preserving expected behavior.
4. Read changes to the tests themselves
Sometimes the implementation needs to change. Sometimes an outdated test needs to change too.
But a green result deserves closer inspection if the agent also:
- Skipped a failing test.
- Removed an important assertion.
- Replaced realistic behavior with a convenient mock.
- Changed the expected result to match the implementation.
- Disabled a check in configuration.
These changes can have valid reasons. Ask for the reason and compare it with the requirement.
“Tests pass” has limited meaning if the relevant checks were weakened.
5. Match verification to the change
Different changes need different evidence.
| Change | Useful verification |
|---|---|
| Calculation | Expected values and boundary cases |
| API endpoint | Status, response body, permissions, and stored changes |
| Database migration | Apply it to representative data and check compatibility |
| UI interaction | Run the interaction and inspect the resulting state |
| Background job | Check retries, duplicate handling, and failure behavior |
A successful build proves that the build completed. It doesn’t prove that checkout works.
A screenshot shows appearance. It doesn’t prove that the button saves anything.
Choose checks that exercise the promised behavior.
6. Look for fixes that moved the problem
A small bug can lead to a surprisingly large patch.
Before merging, ask:
- Did the agent change unrelated files?
- Did it introduce a dependency for a small task?
- Did it suppress an error instead of handling it?
- Did it remove validation to make the request succeed?
- Did it change a shared function and miss its other callers?
Start with the diff summary, then inspect the affected paths.
A large change isn’t automatically wrong. It should have an explanation proportional to its scope.
7. Treat blocked verification as unfinished verification
Sometimes the agent cannot run a check.
The database is unavailable. A dependency cannot be installed. Browser access is missing.
That is useful information.
The report should say:
“Implemented the change, but integration tests could not run because the test database was unavailable.”
Avoid turning “could not test” into “probably works.”
Record the missing check and complete it in the appropriate environment before relying on the result.
A prompt you can reuse
Add this to your next coding task:
Before editing, state the expected behavior and how you will verify it.
After editing, report:
- What changed.
- The exact checks you ran and their results.
- Whether the regression test catches the original failure, if applicable.
- Any tests or checks you changed, removed, or skipped—and why.
- What remains unverified.
Only claim a check passed if you actually ran it and observed a passing result.
This won’t eliminate mistakes. It gives you a clearer starting point for assessing the work.
Before you click merge
Ask one question:
“What evidence shows that this change meets the requirement?”
Sometimes the answer is a focused regression test. Sometimes it is an integration run, a browser interaction, or several checks together.
The important part is being able to connect the evidence to the task.
An AI agent can help you implement and verify a change. Its final message should make that verification inspectable.
What’s the biggest gap you’ve found between an agent saying “done” and the work actually being finished?
Top comments (21)
One thing I’d add is that even a valid test report can miss a production failure. Take a payment service using retries and idempotency keys: the unit tests might pass, but concurrent requests could still create duplicate charges if the database constraint or transaction boundary is wrong. I’d want to see the relevant integration tests against the real persistence layer, not just a green test summary. The agent needs to prove the behavior we care about, not just prove that its code passes the tests it wrote.
Exactly. A green unit test can miss the race that causes the actual problem. For that payment example, I’d want concurrent requests tested against the real database, with one intended charge and a consistent payment record as the outcome.
Right, and I’d also check the failure path, not just the successful concurrent run. For example, if the payment transaction commits but the API times out before returning the response, the client may retry while the original request has already succeeded. The idempotency record, payment state, and retry logic need to agree on the same outcome. Otherwise, you can still get duplicate side effects even when the database ends up with one payment row.
I’d also check what happens when a fix works on one service but breaks a downstream consumer. In an event-driven system, an agent might verify that a Kafka producer publishes the expected event and that the unit tests pass, but never check whether older consumers can still deserialize the new schema or handle out-of-order events. That’s the gap between verifying a change in isolation and verifying its actual impact across the system.
Good point. “The producer publishes successfully” only checks one side. The report should also say which consumers were checked and whether compatibility or event ordering remains unverified.
Exactly, and schema compatibility alone still won’t catch every downstream issue. A consumer might deserialize the event just fine but process it incorrectly because it assumes events always arrive in sequence. I’d test schema changes against the actual consumer versions and replay events with duplicated or delayed delivery. Otherwise, the producer looks healthy while the consumer quietly corrupts the business state.
The 'confirm the regression test fails before the fix' step is the one I'd harden further — an agent can also write a test that only fails against the old code by coincidence, which is why I like your 'read changes to the tests themselves' section as the real backstop. The red-flag list (skipped tests, rewritten assertions, mocks swapped for convenience) deserves its own checklist in the completion report. Do you run the fail-first verification yourself, or does the workflow enforce it automatically?
I wouldn’t call the workflow automatically enforced unless there’s a check outside the agent making that happen. I’d include the fail-first run in the task and ask for its output, then inspect whether it failed for the expected reason. Your point about coincidence matters—a red test alone isn’t enough.
That's the sharper formulation — 'enforced' needs a check outside the agent, not the agent's own report. Including the fail-first run in the task plus inspecting the failure reason closes both gaps: the run proves the test can fail, and the reason check proves it failed for the right cause. Is the reason check itself manual review, or is there a way to make that external check automatic too?
Point 3 is the one I'd underline. A regression test that never failed against the original code proves nothing.
One gap that's easy to miss between "done" and done: tests that didn't actually run. With pytest, for example, if collection fails or the selection matches nothing, you get "no tests ran" and exit code 5, and a summary that says "tests pass" can paper over that completely. So I'd add one item to your completion report: the number of tests executed, not just pass/fail. "0 passed" and "42 passed" both look green in a one-line summary.
Your list of test changes that need a reason (skips, weakened asserts, convenient mocks) also makes a good automated check: diff the tests directory separately and flag any removed assertion for human review.
Agreed—test counts belong in the report. “No tests ran” should never become “tests passed.” I also like checking the test diff separately. Flagging removed assertions gives the reviewer something specific to inspect.
Checking the test diff separately is a good addition. A removed assertion is the quietest way for a suite to go green, and it never shows up in a pass count. Putting it in front of the reviewer as its own line item makes it hard to miss. Thanks Robert!
Defining done before the agent starts is the step most teams skip, and the rest of the post depends on it. Without a written finish line, the agent's summary becomes the definition by default.
Jackson Ly's rootdir story is the clearest example in the thread. Skipped tests going from 61 to 146 with a green run is exactly the kind of evidence that looks fine in a summary and wrong in a diff. Kiell's note on pytest exit code 5 belongs in the same checklist. Launch Gate's idea of keeping acceptance criteria in a file the agent cannot modify is the one I would adopt first, because it removes the quiet option of passing a check by changing it.
On client work we treat the unverified list as part of the delivery, not an apology. If a claim has no evidence attached, it does not count toward done. Has anyone automated that unverified list as a merge blocker, as agentwares suggested?
Section 3 asks you to confirm the regression test fails against the original implementation. That can be made checkable instead of reported: have the agent run the new test file against the pre-change commit in a separate worktree and include the failing output, so the claim arrives with a log. A second gap sits next to "which commands ran", which is where they ran. In my comparisons of agent tools, a working directory under my home folder made Claude Code load my global instruction file into every run, so results depended on a file no report mentioned. Should the completion report also name the directory and environment the checks ran in?
Yes, directory and environment belong in it too. The commit, working directory, runtime version, and loaded config can all affect the result. A separate worktree is useful evidence, provided the test fails because of the original bug rather than a setup problem.
The gap I hit was a green job after a one-line CI edit. It passed a second test path to the same pytest call, so pytest's rootdir moved up to the common parent and the service's pyproject.toml, which set asyncio_mode = auto, stopped applying. Unmarked async tests were skipped instead of run. Every check that ran passed, so a report written to your prompt would probably still have read true. The job went from 451 passed and 61 skipped to 382 passed and 146 skipped.
That’s a great example of a truthful report still giving false confidence. The jump from 61 to 146 skipped tests is the warning the green status hides. I’d add collected, passed, failed, and skipped counts, plus the config file pytest actually loaded.
The retroactive redefinition is the failure mode that ties your five points together: an agent that cannot meet the bar quietly moves the bar - weakens an assertion, swaps in a convenient mock, edits the expected value to match what it built. Two cheap guards on top of your workflow. First, write the acceptance criteria into a file the agent may read but not edit, and diff it at the end of the run; an agent that renegotiates "done" leaves a visible mark. Second, require the completion report to list commands verbatim, because the omissions are as informative as the contents - a suite you expected that is simply never mentioned is a check that was never run.
And "what could not be checked" is the only section of any agent report I trust on first read, since it is the one the agent has no incentive to fabricate.
Point 3 is the one I'd tattoo on the review process: run the new regression test against the ORIGINAL code and watch it fail. It takes two minutes and it kills the entire class of "tests that pass because they assert nothing" — the expect(result).toBeDefined() species.
The test-diff inspection in point 4 matters even more than people think. The scariest agent change I've seen wasn't a skipped test, it was an updated expectation that looked completely reasonable in the diff. Green suite, plausible commit message, and the requirement quietly renegotiated. These days I review test changes in an agent PR with the same suspicion I'd give production changes, sometimes more.
Do you make the fail-on-old-code run part of the task itself, so the agent has to report it? Or is that a step you still do by hand?
I’d make the fail-on-old-code run part of the task and require the output, while still checking why it failed. Your example of a reasonable-looking expectation change is exactly the danger: the suite stays green while the requirement quietly changes. Test edits deserve careful review too.
One addition to point 7: I treat the unverified list as a merge blocker, not just a note in the report. If a check could not run, the change is not ready to rely on, even if everything else passed. The other check I have learned to add is a fresh-environment rerun: a result the agent produced in its own working setup is not the same as the same check passing from a clean checkout or against the live endpoint. That is where I have caught configuration, credentials, and test-data differences that all the local green checks missed.