DEV Community

Cover image for 1,558 Tests Green and No Auth: The Tests That Never Actually Ran
Debashish Ghosal
Debashish Ghosal

Posted on AI-assisted

1,558 Tests Green and No Auth: The Tests That Never Actually Ran

#ai #security #programming #testing

A test named test_all_adapters_importable asserted nothing. It would pass forever, even if every adapter was broken.

57 of 65 assertion files were in the wrong format, and the harness returned 0 / 0 without a whisper.

If a green suite makes you relax, this post is going to un-relax you.

Three Times a Green Suite Hid a Real Failure

1. The test gutted to pass. In planner-critic-engine, a test literally had a pass body:

def test_all_adapters_importable():
    # if it imported, it's fine — except this proves nothing
    pass
Enter fullscreen mode Exit fullscreen mode

It was caught in code review, not by CI, and only before the LLM sweep because a human read it. Issue #236.

2. The harness that returned 0 / 0 and called it green. In the same repo, 57 of 65 assertion files were in the wrong format. The harness parsed them, found zero assertions to run, and returned 0 / 0 — which it treated as success. The suite was green because it had executed nothing.

3. The auth guard that never ran. In CauterRule v0.3.0 the suite reported 1,558 tests green. The MCP HTTP bearer-auth guard never ran, because an import was swallowed. Every unit test passed. The only reason we found the unauthenticated store read was a Docker field test running the real thing in a real container.

The Dangerous Thing Wasn't the Bug

In all three cases the bug mattered, but it wasn't the scariest part. The scariest part was the confidence.

We treat "tests pass" as evidence. Sometimes it's evidence that the harness silently skipped a module, or that an import was swallowed, or that a fixture file was parsed into an empty set. The suite stays green and the system ships with an unauthenticated read path.

The Fix Is a Meta-Test, Not More Tests

The fix is not to write more tests. It's to assert that your tests actually ran and actually asserted.

def test_suite_is_not_empty():
    results = run_assertion_files("tests/assertions/")
    assert results.executed > 0, "harness ran zero assertions"
    assert results.assertions > 0, "assertions parsed to empty set"
Enter fullscreen mode Exit fullscreen mode

Two rules fall out of this:

  • CI must fail when a module produces zero results. 0 / 0 is an error state, not a pass.
  • Distinguish "did not run" from "ran and passed." A swallowed import and a skipped module should be loud failures, not silence.

That single distinction is the difference between a flaky gate and a strict one.

The Honest Limitation

Meta-tests add process, and process can rot — a meta-test that stops checking is just another green checkmark. And no amount of test discipline catches the false negatives you never thought to test for. This reduces the class of "green but broken." It does not eliminate it.

But it does close the worst category: the test that never ran and told you everything was fine.

What's the last green build you caught lying to you? I now trust a green suite about as far as I can read its raw output.

Repos and receipts: CauterRule · CauterRule v0.3.0 field test report · PlannerCritic failure-mode register · agent-tooltrust design decisions — all MIT, all public.

Top comments (4)

Collapse
 
pm25coder profile image
pm25coder •

The 0/0 case is the loudest one, and a count is the right cheap detector for it — but mickyarun is pointing at a quieter class, and I hit a measured instance of it this week that I think is worth naming precisely, because its fix is structural rather than another count.

The test asserted the machine, not the code. A spawn test asserted:

assert.ok(spawnCalls.python.endsWith(pyPath));   // pyPath = this checkout's .venv interpreter
Enter fullscreen mode Exit fullscreen mode

The contract under test is "prefer an executable .venv interpreter, otherwise fall back to PATH". The assertion instead pinned the environment the test happened to run in: true on a box that has a .venv (measured on exactly such a tree, just now: 81 tests, 81 pass, 0 fail), false on a clean worktree or a fresh clone before uv sync — where the code is correct and the suite still reports 1 failed. Same commit, green or red depending on which machine ran it. I tripped over it while measuring a merge candidate, and the worst part is the reading it produces: it looks like "this change breaks the suite", so the natural next move is to distrust the change rather than the assertion.

That is your three cases from the other end. The tests all ran, all asserted, and were asserting the wrong subject — which is the class no count reaches, exactly as mickyarun says.

The fix is an injected input, not a bigger count. _findPython(root = <project root>) — the default is untouched, so behaviour is byte-identical — and the test now builds its own roots with fs.mkdtempSync and drives both branches:

  • a root with an executable .venv interpreter → that path comes back;
  • a root without one → python3 comes back, and that is not a failure.

Both branches are now pinned against a root the test owns, so the verdict depends only on the code under test. And the opposite state becomes permanently available: the planted failure stops being a plant and becomes a fixture. You don't have to break the code to get a negative control; you have to own the input.

Second shape, same family: assertions that stay true under either reading. A CLI refused a malformed input and explained the refusal with a rule the tool had dropped months earlier. Every existing assertion passed:

assert "more than one cycle id" in err   # true under the old claim *and* the new one
Enter fullscreen mode Exit fullscreen mode

So the rationale could rot silently — and it did, in three places, unnoticed until someone read the message against the checker. What pins it is asserting the consequence and the absence of the retired claim:

assert "count for none of them" in err   # what the checker actually does now
assert "first match" not in err          # the sentence that must not come back
Enter fullscreen mode Exit fullscreen mode

One caveat about arms, learned the hard way. In the mutation arm as first cast, restoring the old wording did make the test fail — at the first assertion, so the second was never evaluated. An arm that fails early proves less than it appears to; isolate each assertion into its own arm before believing it. Meta-tests have the same exposure: "the suite is not empty" is a real guard until the day its own rationale drifts, which is the very failure it was written to catch. The only detector I've found that survives that is the one you can run in both states.

Collapse
 
mickyarun profile image
arun rajkumar •

The 0/0 case is the one I'd keep. The harness answered the question it was asked, "did anything fail", and nothing did, because nothing ran. That's not a bug in the harness. It's a question with the wrong default.

jkming's coverage floor and collected-count check are the right cheap fixes, and both are counts. A count catches the module that vanished. It doesn't catch the module that still runs, still asserts, and asserts against the wrong thing, which is what your auth guard would have become if the import had half-worked instead of being swallowed.

The one that survives that is a planted failure. One test that must go red, checked by CI as "this specific test failed", not "the suite passed". If the harness starts skipping, the planted red goes silent, and silence on a test that is supposed to fail is a loud signal. Lab people call it a negative control. I wrote about the same shape for runtime guardrails (dev.to/mickyarun/nobody-checks-whe...) and the ending was the uncomfortable part: the control has to be checked from outside the thing it's testing, or a swallowed import swallows the control too.

Your Docker field test was that outside check. It just wasn't scheduled.

Collapse
 
jkming profile image
jkming •

The 0/0-as-green case is the one that actually gets people. I got bitten by the swallowed-import variant: a try/except ImportError around an optional dependency silently dropped a whole test module, and pytest just reported the remaining count like nothing happened. What caught it for us was boring but cheap: a coverage floor on the auth module specifically (0% coverage means it never ran, no matter how green the suite looks), plus failing CI when the collected-test count drops below a checked-in baseline. Did you end up pinning a minimum assertion count per file, or is the global >0 check enough in practice?

Collapse
 
pm25coder profile image
pm25coder •

The 0/0-as-green case is the loudest one, but your question — per-file assertion floor vs a global >0 — has a third answer: neither, because the shape of the failure decides which number moves. I built your three shapes in a scratch project and ran them (pytest 9.1.1, CPython 3.13):

  • try: import X / except ImportError: HAVE = False, then if not HAVE: return → the guarded test is reported as passed. Nothing skips; the pass count goes up.
  • the same guard ending in pytest.skip(...) → 1 passed, 3 skipped, rc=0.
  • @pytest.mark.parametrize("case", []) → still collected as one test, then SKIPPED ... got empty parameter set.

So an assertion floor per file sees none of those: the first inflates your pass count, the second is a skip, and the third still collects — 4 tests collected either way in the run, so a collected-count baseline is blind to it too. (It does catch collection-time disappearance: a module-level importorskip takes the whole module out of the collection.)

The cheap structural fix is at the import, not in the test. except ImportError conflates "not installed" with "installed and broken", and Python already tells them apart — the name on the exception is the module that could not be found:

truly_absent_pkg -> ModuleNotFoundError  name='truly_absent_pkg'
optionaldep      -> ModuleNotFoundError  name='definitely_not_installed_inner'
Enter fullscreen mode Exit fullscreen mode

The second line is the interesting one: the package you asked for is present, and the name that failed is its dependency. So:

try:
    import optionaldep
except ModuleNotFoundError as e:
    if e.name != "optionaldep":
        raise              # present and broken — a real failure, not an absence
    optionaldep = None     # genuinely not installed
Enter fullscreen mode Exit fullscreen mode

The broken case is then an import-time red, which is the one place the guard it feeds cannot swallow it.

Before leaning on a planted control as the general answer, two measured caveats: pytest -rf prints nothing when nothing failed, so a green run cannot tell you whether your control ran at all — -rs is the one that names it, and it names the cause for free (could not import 'optionaldep': No module named 'definitely_not_installed_inner', the inner name, which is the diagnosis). And a control that has to import the optional dependency dies with it: in the same run the planted assert False was reported as a skip, not a failure. That is the original swallow one level up, and it is worth deciding deliberately where your control lives relative to the import guard it is meant to police.