Would you merge a PR a bot opened? The flaky-or-real check we run first

"I'd never merge a pull request a bot opened." It's the first objection we hear about automated upgrades, and it's the right instinct. The answer isn't trust, it's evidence: you need to know whether a red test was caused by the change at all. Here's the routine, by hand first.

A test goes red on a pull request. The usual responses are "rerun the pipeline" or "start debugging". Both skip a cheaper question: is this failure caused by the change at all?

Step 1: rerun only the failed test

pytest tests/test_checkout.py::test_total -x
npx jest -t "total is correct"

• Fails, then passes: flaky. A timeout, a race, a test that depends on order, real network or the current time.
• Fails twice: probably real. Keep going.

Step 2: run it on main

git stash && git checkout main
pytest tests/test_checkout.py::test_total

If it fails on main too, your change didn't break it. It was already broken, and that's a separate ticket, not a reason to block this pull request.

Step 3: only now, debug

At this point you know the failure is real and new. That's the only kind worth blocking a merge on.

Why teams skip this

CI shows one colour per run. So the habits are "rerun until green", which hides real failures that happen to pass on a retry, or "block on red", which lets one flaky test hold up every pull request for months.

Baking it into CI

pytest can retry failures and report them separately:

pip install pytest-rerunfailures
pytest --reruns 1

A test that only passes on the rerun is flagged in the output, so it can be tracked instead of forgotten.

The stronger version compares results before and after the change: the same tests run on the base commit and on the pull request, and only new failures count. That's how Versyka decides whether an automated upgrade is safe: tests that were already failing are listed in the pull request, a test that fails once and passes on retry is labelled flaky, and only a new, repeatable failure stops the pull request from opening.

What would you need to see in an automated pull request before you'd merge it?