The suite said 3 failed, 2 passed. The test I was staring at had passed ten minutes earlier, alone, in 0.01 seconds. Same file, same code, same machine. I changed nothing between the two runs except how many tests I asked pytest to run.
That is the kind of bug that makes you distrust your own eyes. The test was right. The code was right. The suite was wrong, or the suite was right and the single run was lying. One of those had to give, and the answer turned out to be a single line in a completely different test file, a line that passed its own test and poisoned everything after it.
I built the whole thing from scratch to catch it cleanly: a tiny pricing module, five tests, pytest 9.2.0 on Python 3.12.3, every run saved. I even tripped over the setup first. The system Python on this machine refused pip install pytest outright, PEP 668, externally managed environment, and told me to use a virtual environment instead. So I did. That is already a small lesson about test setups: the environment you run in is part of the experiment, and mine started by saying no.
The code under test
Fourteen lines, nothing clever:
"""Tiny pricing module. DISCOUNT is module-level shared state."""
DISCOUNT = 0.0
def total(prices):
return sum(prices) * (1 - DISCOUNT)
One module-level variable, DISCOUNT, starting at 0.0. total() adds up the prices and applies it. The suite had five tests across two files. test_polluter.py held one test, a half-price sale check. test_victim.py held four: full-price total of [100, 100], single item [50], empty cart [], and three items [10, 20, 30]. Expected results: 200, 50, 0, 60. Simple arithmetic, the kind you would bet money on.
The polluter test looked innocent too:
import pricing
def test_half_price_sale():
# POISON: mutates shared module state and never restores it.
pricing.DISCOUNT = 0.5
assert pricing.total([100, 100]) == 100
It sets the discount to 0.5, checks that the total halves to 100, and passes. What it never does is set the discount back. That missing line is the entire article.
Run one: the full suite
test_polluter.py::test_half_price_sale PASSED [ 20%]
test_victim.py::test_full_price_total FAILED [ 40%]
test_victim.py::test_single_item_total FAILED [ 60%]
test_victim.py::test_empty_cart_total PASSED [ 80%]
test_victim.py::test_three_items_total FAILED [100%]
========================= 3 failed, 2 passed in 0.05s ==========================
Three failed, two passed, in 0.05 seconds. I had expected one failure when I wrote the suite. I got three, and the honest thing is to say so: my guess was wrong, the run was right. The failures were loud and specific:
> assert pricing.total([100, 100]) == 200
E assert 100.0 == 200
> assert pricing.total([50]) == 50
E assert 25.0 == 50
> assert pricing.total([10, 20, 30]) == 60
E assert 30.0 == 60
Every total came back exactly half of what it should be. 100 instead of 200. 25 instead of 50. 30 instead of 60. Halved, all of them, like a 50% sale was still running. And one test passed in the middle of the wreckage: the empty cart. total([]) is 0 with any discount, because zero halved is still zero. That passing test is the sneaky part. If the suite had only the empty-cart check after the polluter, everything would have looked green and the poison would have shipped.
Run two: the victim alone
I ran the failing test by itself:
$ python -m pytest test_victim.py::test_full_price_total -v
test_victim.py::test_full_price_total PASSED [100%]
============================== 1 passed in 0.01s ===============================
Passed. In 0.01 seconds. The test was fine. The code was fine. The failure only existed when other tests ran first. That is the signature of shared-state pollution: the test is not broken, the world it runs in is different depending on what ran before it.
Run three: polluter plus victim
Two tests, in the order pytest collects them:
test_polluter.py::test_half_price_sale PASSED [ 50%]
test_victim.py::test_full_price_total FAILED [100%]
========================= 1 failed, 1 passed in 0.04s ==========================
There it is, with nothing else in the room. The polluter passes, the victim fails, and the only thing connecting them is a module they both import. pytest runs files alphabetically, so test_polluter.py sorts before test_victim.py, and the discount is still 0.5 when the victim's assertions execute. The victim never touches DISCOUNT. It never imports anything odd. It just calls total() and trusts the module to be in its starting state. That trust is the bug.
The one line
pricing.DISCOUNT = 0.5
That is the polluting line, the whole of it. One assignment, no cleanup. The test that contains it passes, which is why nobody catches it in review. A test that passes is a test nobody looks at. The damage shows up somewhere else, in a file the author of the polluter may never have opened, as a failure that looks like a pricing bug. I have watched people debug the wrong file for an hour on exactly this shape of failure: the arithmetic is right, the test is right, so they start questioning the framework, the runner, the cache. The framework is fine. A previous test left the room dirty.
Why does this keep happening in real codebases? Because module-level state is the easiest thing in the world to reach for. A config flag, a cache dict, a "current user" global, a feature toggle set in one test to check a branch. The test sets it, asserts, moves on. The next test inherits a world the author never described. Multiply that by a few hundred tests and the suite becomes order-dependent: green in one order, red in another, and nobody can say which order is true. The test-verification cluster on dev.to this week, 42 posts under "Verifying AI Agent Patch Test Results" plus xulingfeng's AI-testing post at 132 reactions and 76 comments, is full of people discovering that a green suite can be a lie. A suite that passes because of the order it ran in is not verified. It is lucky.
The fix
One finally block:
import pricing
def test_half_price_sale():
pricing.DISCOUNT = 0.5
try:
assert pricing.total([100, 100]) == 100
finally:
pricing.DISCOUNT = 0.0 # THE FIX: restore shared state
The finally matters. A plain reset line after the assert would be skipped if the assert failed, leaving the poison in place on the exact run where you most need a clean room. finally runs either way. Then the full suite:
test_polluter.py::test_half_price_sale PASSED [ 20%]
test_victim.py::test_full_price_total PASSED [ 40%]
test_victim.py::test_single_item_total PASSED [ 60%]
test_victim.py::test_empty_cart_total PASSED [ 80%]
test_victim.py::test_three_items_total PASSED [100%]
============================== 5 passed in 0.01s ===============================
Five passed in 0.01 seconds. Same tests, same code under test, one line of cleanup added. The suite went from 3 failed to zero without touching pricing.py at all. The production code was never broken. The tests were poisoning each other.
What I check for now
Three habits, all cheap. First, any test that mutates shared state must restore it in a finally, or better, through pytest's monkeypatch fixture, which undoes the change automatically when the test ends. monkeypatch.setattr(pricing, "DISCOUNT", 0.5) does the same job and cannot forget the cleanup, because the fixture owns it. Second, I run a failing test alone before I debug it. If it passes alone, I am not looking at a code bug, I am looking at a neighbour. Third, I run the suite in a shuffled order once in a while. If the results change with the order, the suite has pollution hiding somewhere, and the shuffle just told me where to look.
The deeper habit is the one this experiment forced on me: treat a green suite as a claim, not a fact. Five tests passing in 0.01 seconds felt like proof until three of them failed for a reason that had nothing to do with them. The fix was not in the failing test. It was never in the failing test. It was one line in the test that passed.
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam
Top comments (0)