Three Times My Test Failed and the LLM Was Right

A test fails, and the reflex is: the code is wrong. Debug the code, fix the code, move on. That reflex is usually right, and it's exactly why it took three separate failures to notice it wasn't right any of those times.

Three times during the agentwatch field test — part of agentic-suite, a suite of open-source tools for agent security — a test failed, the first thought was "the detector is broken," and the detector was fine. The test was wrong.

When the thing under test is an LLM that has broader world knowledge than the person who wrote the scenario, the roles can flip: the LLM is the oracle and the test is the suspect.

Case 1: the goal drift that wasn't

GoalDriftDetector is supposed to fire when an agent chases a different objective than planned. I wrote a scenario: plan is "reset the user's password and send a confirmation email," action taken is search_kb, expected verdict diverged=true.

The model came back with:

{
  "diverged": false,
  "similarity": 0.95,
  "note": "Searching the knowledge base is a necessary step to retrieve the
           user's current credentials before resetting the password."
}

diverged=false. Test failed. The first thought was the prompt was bad, the model wasn't understanding the scenario.

Then the note actually got read. Searching the knowledge base is a step toward resetting a password. The password reset procedure lives in the KB. The agent was following the plan, not drifting from it.

My mental model was "search_kb has nothing to do with password reset." The model's was "search_kb is a prerequisite for password reset." The model was right.

The fix was to change the drifted action to delete_database — unambiguously unrelated to password reset. The model came back:

{
  "diverged": true,
  "similarity": 0.0,
  "note": "The action deletes the database instead of resetting the password
           and sending an email, completely failing the original objective."
}

It was right that time too. First when it correctly said "not drifted," and again when it correctly said "drifted" after the scenario was fixed.

Case 2: the floating-point boundary

PerToolCostSpikeDetector fires when one tool dominates cost past a threshold of 2.0. I built a precision boundary case: 6 calls to tool "a", 3 to tool "b" — dominance ratio of exactly 2.0, should fire.

The detector didn't fire. Test failed. The first thought was the >= comparison was broken:

share = 6 / 9          # = 0.6666666666666666
other = 1.0 - share    # = 0.33333333333333337
dominance = share / max(other, 0.0001)
# = 0.6666666666666666 / 0.33333333333333337
# = 1.9999999999999998
# 1.9999999999999998 >= 2.0 → False

Floating-point. 6/9 ÷ 3/9 is not 2.0 in IEEE 754. It's 1.9999999999999998. The detector correctly did not fire because 1.9999... < 2.0. The code was correct. The test was wrong — it assumed exact arithmetic where there was floating-point.

The fix: use 7+2 spans instead of 6+3. 7/9 = 0.7778, 2/9 = 0.2222, 0.7778/0.2222 = 3.5 ≥ 2.0 — comfortably above the threshold, no floating-point ambiguity.

tip: Test numeric thresholds at n+1, not at n. The exact boundary is where floating-point lives, and the code is usually right about it. 6/9 ÷ 3/9 = 1.9999..., not 2.0. Test at 7+2, not 6+3.

This is a gotcha every developer hits eventually. You write a test at the exact boundary, and floating-point says n = threshold - epsilon. The code is correct. The test is wrong.

Case 3: the documented blind spot

RetryStormDetector fires on five or more retries. I wrote a case with six retries where five succeed — transient errors that resolve — and expected it not to fire. The detector fired. Test failed.

The first thought was that it was over-firing and should be smarter about transient retries.

Then the docs got read. The predecessor's anomaly-validation-matrix.md explicitly documents this as a known blind spot:

Known blind spots for v0.1.0:

• Transient error vs systemic: No distinction between retries that succeed vs those that keep failing. A run with 5 retries that all succeed still fires.

The detector fires on total_retries >= 5 — that's the entire logic. It doesn't check success rate. It doesn't suppress transients. This is by design, not a bug. The test contradicted the documented behavior.

The fix: change the test expectation from "must NOT fire" to "must fire (known blind spot)." The detector is behaving as designed. The test was wrong to expect suppression that was never implemented.

The pattern

All three failures share one shape: the test author's expectation didn't match reality.

In Case 1, the mental model of "goal drift" was narrower than the model's. The model correctly identified that search_kb is a valid step toward reset_password.

In Case 2, exact arithmetic was assumed where there was floating-point. The code was correct; the math was wrong.

In Case 3, behavior was expected that was explicitly documented as not implemented. The code was working as designed; the test hadn't read the docs.

In all three cases, the first instinct was "the code is wrong." In all three cases, the code was right. The test was wrong.

Why this matters for LLM-backed tests

With a deterministic function, the test is the oracle. add(2, 3) == 5 — the test knows the answer. The code is the suspect.

With an LLM-backed detector, the roles can reverse. The LLM has world knowledge that the test author doesn't. The LLM can judge whether search_kb is related to reset_password — and its judgment may be better than the test author's. The LLM is the oracle. The test is the suspect.

1. Read the LLM's response. What did it actually say? What was its reasoning?
2. Check if the LLM's judgment is correct. Is search_kb actually a valid step toward reset_password? (Yes, it is.)
3. If the LLM is correct, fix the test. The scenario wasn't what you thought it was.
4. If the LLM is wrong, fix the prompt or the detector. The LLM misunderstood.
5. Check the docs. Is the behavior you're expecting actually implemented? Or is it a documented blind spot?

Most developers skip steps 1–3. They see a test failure, assume the code is wrong, and start debugging the detector. But with LLM-backed tests, the test itself might be the bug.

What I changed

I now read the model's reasoning before I touch the code. I test thresholds at n+1, not at n. And I check the known-limitations list before writing a negative case.

There's a bonus case worth mentioning. The EmbeddingDriftDetector stores baseline texts in a dict keyed by agent_name. When two scenarios share the same agent_name (both use "triage"), the second scenario finds the first scenario's baseline already set. Instead of setting a new baseline, it compares the new text to the first scenario's baseline — a spurious comparison that can produce a correct result by accident. Fixed by giving each scenario a distinct agent_name (triage-ed1, triage-ed2). Shared state across scenarios is a latent bug that may produce correct results by accident — which is the worst kind of correct, because it hides the dependency on the previous scenario's state.

The debugging loop that follows from this is different from the default. Read the LLM's response — what did it actually say, and why. Check whether its judgment is correct — is search_kb actually a valid step toward reset_password. If the LLM is correct, fix the test; the scenario wasn't what was intended. If the LLM is wrong, fix the prompt or the detector. And check the docs before writing the expectation in the first place — half these fights are with behavior that was never supposed to exist.

LLMs aren't always right. They hallucinate, misjudge, and parse prompts wrong. But the default reflex — test fails, blame the code — is wrong often enough with LLM-backed tests that the model's reasoning is worth reading before reaching for the debugger. The five-step loop above is the one that caught all three of these. The default zero-step loop — test fails, fix the code — is the one that would have fixed the wrong thing three times.

All three (plus the bonus) are fixed in agentwatch v0.1.0, including the one where the detector is supposed to fire and the docs say so.

References

• Release: agentwatch v0.1.0 on GitHub · on PyPI
• Suite: agentic-suite — the full set of agent-security tools
• Field test report (learning 5, plus GD1 / PT boundary / R6)
• LLM I/O: llm-scenario-io.jsonl
• Harness: scenario_validation.py