The Second Case Was Not the Second Case. I Passed pytest a Set.

Have you ever rerun pytest --lf and then watched a completely different case fail instead? I had that sinking feeling this week, and the confusion lasted longer than I want to admit. The suite was still green on my laptop, yet the remote collection order refused to match mine. Was pytest broken, or had I handed it an unordered collection and then trusted the labels?

This is a field note, not a victory lap, because I chased three other theories first. I blamed xdist, then the cache plugin, then a stale .pytest_cache directory on disk. None of those were innocent, but none of them were the first cause either. The parametrize decorator had received a set, and Python was free to shuffle it.

What I thought was happening

Why would two processes collect the same file and then print a completely different order? I assumed a plugin was injecting extra tests, because that story had bitten me before. I also assumed the remote run used a different pytest.ini, because configuration discovery is a maze. I opened the xml report and counted testcase elements like that would reveal a missing module.

The report revealed nothing useful except my talent for reading the wrong column first. The generated test looked reasonable at a glance, which is how mistakes like this get committed. Unique string cases lived in a set, then landed in parametrize beside a parallel list of labels. Do you see the trap already, or did you also trust that uniqueness implies stability?

Forty-eight hours, written down

I am writing this in the order I actually worked, including every dead end I hit. That is the only way these notes stay useful after the adrenaline leaves the room. If you want the tidy version, skip to the artifact and steal the helper. If you want the bruises, the numbered log below is the honest timeline.

1. During the first six hours I deleted .pytest_cache and reran with --cache-clear, yet collection still drifted.
2. During hours six through fourteen I pinned pytest and pluggy, then compared pytest --fixtures across both processes.
3. No extra plugin appeared, which should have been a clue that I was auditing the wrong layer.
4. During hours fourteen through twenty-six I exported PYTEST_ADDOPTS and searched for an -n auto that was not there.
5. During hours twenty-six through thirty-six I sorted junit XML by classname; names matched and document order did not.
6. During hours thirty-six through forty-eight I printed repr(cases) from a session fixture, and the set had moved.

I would still clear the cache once, because cache lies are real, just not this time. I would still audit plugins once, with --trace-config, and then I would stop repeating it. The session fixture print is the step I should have done on hour two instead. Everything else was motion, and motion is how a two-hour bug becomes a forty-eight hour story.

The artifact: a tiny suite that shuffles on purpose

The snippets below are a reconstructed demonstration that you can run on your own machine tonight. They match the failure mode I hit, without dragging in the rest of a private product suite. Please treat them as a lab, not as a claim that your currencies are broken. If the two collect-only dumps disagree, you have reproduced the entire investigation in miniature.

Create test_currency_labels.py:

import os
import pytest

# Unordered on purpose. Do not copy this pattern into production tests.
CASES = {"usd", "eur", "gbp", "jpy"}
LABELS = ["americas", "europe_a", "europe_b", "asia"]

@pytest.mark.parametrize("code", CASES, ids=LABELS)
def test_currency_code_is_alpha(code):
    assert code.isalpha()
    assert len(code) == 3


def test_hash_seed_is_visible():
    seed = os.environ.get("PYTHONHASHSEED", "<unset>")
    # Visible in `-s` output when you are comparing two processes.
    assert seed is not None
    print(f"PYTHONHASHSEED={seed}")

Now collect twice with different hash seeds, and use two real processes rather than a loop in one interpreter. You want two processes, because hash randomization is chosen at startup, not at import of your test module. If you reuse one Python process, you will keep confirming the order you already believe in. Does that sound obvious now, or did you also rerun the same shell and call it a second sample?

python -m venv .venv
source .venv/bin/activate
pip install "pytest>=8"

PYTHONHASHSEED=1 pytest --collect-only -q test_currency_labels.py
PYTHONHASHSEED=2 pytest --collect-only -q test_currency_labels.py

When I ran this lab, one seed paired usd with a different label than the next seed. I will not hard-code that pairing here, because it depends on the interpreter you exec. If you xfail test_currency_code_is_alpha[asia], which currency did you actually skip in that run? The label moved with the iteration order, and last-failed then defended a story about the label.

Here is a small helper I now keep beside tox.ini, and it is deliberately boring. Boring is the point, because I do not want another clever plugin in this path. Run it under several seeds whenever an xfail talks about a label instead of a value.

# tools/show_param_pairs.py
"""Print parametrize pairings for one file under several PYTHONHASHSEED values."""
import os
import subprocess
import sys

SEEDS = ["0", "1", "2", "3"]
TARGET = sys.argv[1] if len(sys.argv) > 1 else "test_currency_labels.py"

for seed in SEEDS:
    env = os.environ.copy()
    env["PYTHONHASHSEED"] = seed
    print(f"\n=== PYTHONHASHSEED={seed} ===")
    subprocess.run(
        [sys.executable, "-m", "pytest", "--collect-only", "-q", TARGET],
        env=env,
        check=False,
    )

Run it with python tools/show_param_pairs.py test_currency_labels.py and then read the blocks slowly. If the blocks disagree, you do not have a flake in the assertion on that line. You have an unordered iterable feeding labels that are stored by position, which is a different bug.

A decision table I needed on hour one

I needed a table on hour one, before I had opinions about pytest-cache or remote runners. The middle columns are the only questions that mattered once I stopped changing fixtures at random. If the left column says set, I stop arguing with the log and I fix collection first.

Value-based ids such as [usd] stay honest even when the print order changes under you. Positional ids such as americas stay pretty, and pretty is how I xfailed the wrong currency. I still like pretty reports, but I now bind pretty with pytest.param so the id travels.

Where a second process actually helped

I needed a Python interpreter that was not the one inside my already confused laptop session. Local reruns with the same seed hid the shuffle, because I kept confirming my own collection order. A second process is the cheapest way to let PYTHONHASHSEED surprise you on purpose today.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to walk through two collect-only dumps that refused to match. I used the free server option to run pytest --collect-only in a process that was not mine. That pairing mattered because the model could only see the file I pasted into the session.

The server actually executed a different hash seed, which is the part a chat window cannot fake. I still pasted the collect-only output back, and I still looked at the ids myself afterward. Did the model invent PYTHONHASHSEED as the root cause, or did I ask a leading question? I asked why two dumps disagreed, then I verified the environment variable with the helper above.

I am not going to pretend this replaces a proper test matrix in CI, because it does not. It did give me a second process on the same afternoon, which was all I needed then. That was enough to stop blaming pytest-cache for a set I had written with my own hands.

What broke while I was "fixing" it

I tried sorted(CASES) and then forgot that ids=LABELS was still a positional list underneath. Sorted values plus stale labels is a quieter lie, and quieter lies survive code review longer. Have you ever congratulated yourself for sorting, then committed the same ids array without re-reading it? I have, and the collect-only dump is what made me put the commit back down.

I tried PYTHONHASHSEED=0 in my shell profile, which stabilized tests and would have leaked into other apps. Disabling hash randomization in a shared profile is a gift to anyone who sends colliding keys. I tried deleting labels entirely, which made node ids honest and made the HTML report slightly uglier. Ugly and honest won, because I can live with [usd] in a report I actually trust.

I also asked the coding model to make the tests deterministic, without showing any collect-only output. It rewrote the assertions, because that is what models do when you mention failure without evidence. If you do not paste the collection dump, you will get a better assert, not a better iterable. Would you like a prettier assertion, or would you like the label to stay glued to usd?

What I would repeat

These are the habits I am keeping, and they are smaller than the investigation that produced them. I do not need another plugin; I need parametrize inputs that still make sense after a fork. The list is the whole practice, and I will ignore it in a month unless it lives here.

• I keep parametrize inputs as lists or tuples, even when the domain feels like unique currency codes.
• I bind pytest.param(value, id=label) so the label travels with the value, not with the index.
• I collect under at least two PYTHONHASHSEED values before I trust an xfail that names a label.
• I paste --collect-only -q output into a coding session when the complaint is order, not logic.
• I leave PYTHONHASHSEED unset in application runtimes, and I pin it only inside the test runner when debugging.

Would I pin the seed in pytest.ini as a default for every developer on the repo? I would not, because a pinned seed hides the bug until somebody else runs the file. I would rather the suite be order-safe than seed-safe, even if the report looks a little plainer. A seed is a flashlight, and a flashlight is not the same thing as a lock.

Limitations, and who should skip this

This writeup is about collection order, not about finding assertion bugs in business logic faster. If your parametrize inputs are already tuples with value-based ids, you can ignore the whole story. If you do not use pytest, none of these commands will help you, and that is fine. If you need hash randomization to match production, do not set PYTHONHASHSEED=0 in a global profile.

The extra server path is a convenience process, not an evidence locker for a compliance audit. I did not record hardware, quotas, or model names here, because I did not measure those things. Treat any remote shell as untrusted for secrets, even when you are only debugging collection order. Do not paste .env files into a coding session because two dumps of node ids disagreed today.

This also does not catch unordered iteration inside the production code that sits under the test. A set shuffle in your handler can still pass a parametrize suite that I just made stable. That is a different bug, and it needs a different fixture than the one in this note.

Notes I am taking into the next suite

I keep a one-page checklist now, because I will forget this story in about a month. The questions are short on purpose, so I will actually ask them before I rerun --lf. If item three fails, I stop debugging the assertion and I stare at collection until it agrees.

1. Is the first argument to parametrize a list, a tuple, or an accident that used curly braces?
2. Does every ids entry come from the value itself, rather than from enumerate or a parallel list?
3. Do collect-only dumps under PYTHONHASHSEED=1 and PYTHONHASHSEED=2 match each other exactly, line for line?
4. Did I xfail a label that might move, or a value that cannot move without changing the test name?

I reran collection on a second process so I would stop gaslighting myself with one laptop order. If you already have CI workers, use those workers and skip standing up any extra shell. A second interpreter is the whole trick, and it does not require a new pytest plugin. The checklist above is what I would actually repeat, and the rest was expensive curiosity.