Result first: an agent made 16 commits, and somewhere along the way add 2 3 started returning -1. I gave git bisect one good commit, one bad commit and a 3-line test script. It ran the test 4 times and named the first bad commit, "commit 11", which is exactly the one where the bug was planted.
printf '#!/usr/bin/env bash\nsource ./calc.sh\n[ "$(add 2 3)" = 5 ]\n' > ../test.sh && chmod +x ../test.sh
git bisect start HEAD <known-good-commit>
git bisect run ../test.sh
git bisect reset # go back to where you started
The rules for the test script (from the git docs): it "should exit with code 0 if the current source code is good/old, and exit with a code between 1 and 127 (inclusive), except 125, if the current source code is bad/new." Exit code 125 means "this commit can't be tested, skip it." "Any other exit code will abort the bisect process."
Why this suits agent workflows:
- Agents make many small commits, and bisect is a binary search, so the number of test runs grows slowly as the history grows (here 4 runs for 16 commits).
- Keep the test script outside the repo or untracked, as
../test.shis here, so checking out old commits doesn't change the test itself. - Use exit 125 for commits that don't build, rather than marking them bad.
Do this now: next time an agent session ends with a regression, write the one-line check first, then run git bisect run before reading any diffs.
Sources
- git documentation, git-bisect: https://git-scm.com/docs/git-bisect
From Penloom Studio's free AI lessons: penloomstudio.com
Top comments (1)
Love that you kept the test script outside the repo. It's easy to miss, and an agent that 'fixes' the test along with the code can send bisect off blaming the wrong commit. One more trick for agent histories: if the agent merged side branches,
git bisect start --first-parentkeeps you on the mainline so you land on the merge you actually shipped. Did the agent's commits build cleanly often enough that exit 125 rarely came up?