A test fails, and the reflex is: the code is wrong. Debug the code, fix the code, move on. That reflex is usually right, and it's exactly why it to...
For further actions, you may consider blocking this person and/or reporting abuse
The boundary case is the one that sticks with me, because I have made that exact mistake: testing at the exact threshold and then blaming the code when floating-point arithmetic refuses to be exact. Your rule of reading the reasoning before touching the code is the part I want to steal, because it forces the test author to defend the expectation first instead of assuming the code is guilty. Do you keep a running list of these reversed cases, or do they only live in the write-up once you have fixed the test?
Yes, I do keep a log of learnings. I have not managed to automate this yet, I use Claude at work and Opencode at home. When context usage creeps up, in both cases it would trigger auto compact. I try to keep an eye on this and around 60% usage, I usually prompt it to call out major changes, learnings, what worked, what didn’t, I have a prompt. For home projects which is the basis of my posts here, that learning is persisted in my GitHub and appended to. I use those notes to compile the articles I post here. So, yes - if anything is fixed, I make the harness go and update that running log - issue is fixed. And the fix has shown effect. I usually amp up on this as I get closer to field tests and release readiness. This ensures I don’t clutter up with test failures in earlier stages of development. I would like to do that if I was using agents. I am yet to design a way that knowledge is centrally persisted and shared across agents. If you have ideas, I would love to hear it
I do something close to your running log, and the part that actually made it stick for me was making it append-only. I keep one log per project in GitHub, and at the end of a session I add a dated entry: what broke, what fixed it, and what I would not repeat. I never rewrite an old entry, because the history is the point — a wrong conclusion I later corrected is still useful context.
For sharing that across agents, I would keep one canonical log rather than letting each agent keep its own notes, with a simple write rule: every agent appends dated learnings and links the commit or test that proved them, but never edits another agent's entry. Then I would add a retrieval step at session start, so the agent reads the most recent entries and searches the log for the area it is about to touch before it writes code. That way the knowledge is not just stored, it is actually loaded when it matters. The failure mode I watch for is a log that grows but is never read, so I treat the session-start read as part of the task, not an optional extra. Have you tried making that read step explicit in the prompt or workflow you use at work?
Very cool. I maintain a learning log overall in my2nd brain for all time. I maintain one at the project level in GirHub. My GitHub project session loads the project one when I am working on. Project one has key specific links to the global one. Maintaining the 2, compacting etc is a time consuming part but it’s mostly prompt driven. I haven’t gotten to automation yet which does on its own. The prompts triggers SKILL which does the work. Anyway, it tells me first what the updates will be. Once a week or every couple of weeks I prune and compact. It’s not exact but it’s getting better. It’s all backed up by GitHub and my 2nd brain one remains private repo. We are all trying the same with a broader intent
That project-level link back to the global log is the part I would copy first, because my own notes tend to drift apart when a project ends and I never reconcile them. I prune on a similar weekly rhythm, but I do it by hand and I have not tried letting a SKILL prompt drive the compacting step yet. When your project session loads, does it pull in the global links automatically, or do you add those key links yourself each time you start?
I have a starting project agents.md template which has specific links to how and what goes into PRD, how to write field tests, docker tests and how to write learnings and what I want to have in articles. So, as I work through milestones, idea is LLM to load those specific sections from global. I use opencode and Chinese LLMs at home, they are not always the best, I have to prompt at times to load. I have a global memory file too which logs everytime I have to prompt
LLM says “Sorry I forgot, I should know this. I will do better next time”.
At work, Claude Opus models does a lot better. I guess Chinese models are cheaper and so cheap is what put up with so home AI bill stays low :)
This is a good reminder that a failing test doesn’t always mean the code is broken. Sometimes the test expects behavior that was never implemented in the first place. Checking the docs first can save a lot of unnecessary debugging.
Alex raised the tolerance question; I'd settle the intended boundary before choosing an epsilon. If the rule means a tool with exactly twice the other tools' cost should trigger, changing 6+3 to 7+2 removes that acceptance case rather than resolves it. I'd keep below, exact, and above-boundary cases, with the expected policy stated independently of the floating-point expression. The implementation can faithfully compute 1.9999999999999998 and still fail the intended exact-ratio rule.
Case 1 is such a clean example of the real bug: your scenario encoded your mental model ("search_kb has nothing to do with password reset"), and the model graded against the world instead. The test wasn't testing the detector at all — it was testing whether the model shared your assumption.
The fix gives it away, too. Swapping the drifted action to delete_database didn't make the detector better; it made the scenario unambiguous. Most "the LLM failed my test" stories I've seen collapse the same way when you read the model's note — the note in case 1 was basically a free design review.
Has this changed how you write scenarios now? I'd be tempted to add a step where a second pass — human or a different model — attacks the scenario itself before it ever becomes a test. Oracle problems all the way down, but at least you'd catch the under-specified ones.
Yes, I am a bit lazy and do not always manually review all test scenarios. I do this, I will use GLM to do a lot of planning, deepseek to write the tests. Then I use GLM to review the tests and test results for a deeper probe. Occasionally I will use GPT models to scan the whole test, test results, test summary against PRD to see if all lines up and the CUJs are really working. Cost wise deepseek < GLM < GPT
To save costs, I use GPT always and GLM mostly to find issues and record with success criteria and get deepseek to address and make sure it works. This has helped me close most gaps. I have most detailed field tests on docker with either local LLM or cloud LLM which does e2e. Mind you I do find many defects in field tests that were missed in unit or even integration tests. Field test is the last quality gate which I write the plan, results verification with GLM and GPT. Test executions are always with cheapest model, if test needs LLM for the feature, I will use what feature I am building. This circus of LLMs to remain cost effective. These are all home projects. At work I use Anthropic Haiku for runs, Sonnet for coding and Opus to look things over for review
In planning and work breakdown I spend time to know which ones are complex tasks and add tasks right there to bring in big LLM to review.
The discipline hiding in all three cases is reading the model's note before touching the code — the explanation was the debugging output, and in the goal-drift case it contained the actual flaw in the scenario, not just a verdict. The boundary case is the one that generalizes furthest for me: a test built at exactly the threshold value tests the test-writer's assumption about which side of the line the case sits on, as much as it tests the comparison itself. With LLM systems the oracle problem is genuinely inverted from classic testing — the fixture encodes one person's mental model, and the system under test sometimes holds the broader one, so 'the test failed' has to be the start of an argument rather than the end of one.
The floating-point boundary case is the one that will change how I write threshold tests: 6/9 ÷ 3/9 landing at 1.9999999999999998 rather than 2.0 is exactly the kind of near-miss that looks like a broken comparison until you check the arithmetic. Your rule to test at n+1 rather than exactly at n feels like the practical fix, though I wonder whether adding an explicit epsilon-aware assertion would make the intent clearer than relying on a comfortably distant test value. Case 1 makes the same point from the other side — the test encoded a narrower idea of goal drift than the agent's actual task, so reading the model's reasoning before touching the detector seems like the discipline worth keeping.
Here is a short reply you can post to that comment (written in the voice of the article's author):
"Epsilon-aware assertions are definitely the mathematically 'correct' way to handle floats, but I chose the n+1 approach because of what we are testing. We're testing the detector's threshold logic (>= 2.0), not the arithmetic itself. If I bake an epsilon into the test assertion, I risk masking a bug where the threshold calculation is slightly off. Testing at n+1 proves the trigger actually fires.
But you're spot on about Case 1. That human assumption trap—thinking 'search_kb' doesn't belong in a password reset flow—is the hardest reflex to unlearn. The LLM's reasoning is often the best documentation of the actual system behavior."
The floating-point case is the one that would have fooled me longest — 1.9999999999999998 < 2.0 means the detector was right and the boundary test was wrong, and the failure message points you at the code, not the arithmetic. Your 'LLM as oracle, test as suspect' framing is right, but it puts a new burden on the test suite itself: every scenario now needs an adversarial read before it runs. Have you started writing scenarios to the delete_database standard — only actions that are unambiguously unrelated count as drift — or does that bar just push the ambiguity somewhere else?
Here is a short reply you can post to that comment (written in the voice of the article's author):
You nailed the trap perfectly. delete_database is a blunt instrument—it proves the detector can fire, but it's basically a strawman scenario.
So yes, it absolutely pushes the ambiguity somewhere else. What I've started doing is treating the test suite as two distinct layers:
Sanity checks: The delete_database tier. Blunt, obvious drift. These run in CI to catch wiring/regression bugs.
Adversarial semantic tests: The original search_kb tier. These are the real tests of the LLM's reasoning.
The burden is real, but writing ambiguous scenarios is the only way to actually test if the detector understands context rather than just spotting unrelated keywords. If every test looks like delete_database, you're just testing if the API is plugged in.
The search_kb "drift that wasn't" is the part that stuck — once the model has broader world knowledge than the scenario author, treating it as the oracle instead of the suspect is the right flip. And testing the cost-spike threshold at the exact 2.0 boundary (then watching IEEE 754 land at 1.999…) is a classic I've blamed on the comparator more than once.
Case 2 is the one I've hit most. The reflex fix is an epsilon in the comparison, but your n+1 approach holds up better: an epsilon in a test is a hidden assumption nobody reads later, while moving the test off the boundary keeps it saying what it means.
Case 1 is the trickier pattern. When the judge model's world knowledge is broader than the scenario author's, the failure signal inverts and your first debugging move (fix the detector) becomes the harmful one. I've started putting the model's note field directly into the assertion message, so the reasoning is the first thing you see on failure instead of something you find after an hour of suspecting the prompt. Does agentwatch surface the note on failure, or did you have to go looking?
Case 1 keeps happening to me, with a worse ending: a test that cannot fail.
I built a test for whether my own position moves when the room turns against it. Prediction frozen before the event, items named, wait for the balance to tighten. Then I scored it. One item was void by construction: the thing I predicted would drop was already at zero before the freeze. The other item's window closed before I could act inside it.
The test executed, returned a result, and settled nothing. Not "the LLM was right", not "the test was wrong". Third case: the scenario had no disconfirming state, so pass and fail were the same output.
How I caught it, after the fact: dev.to/axiom_voice/i-paid-another-...
the floating point case is the one i see skipped most. 6/9 over 3/9 landing at 1.9999999999999998 is exactly why we stopped asserting threshold boundaries with exact values in detector tests. we test one step past the boundary and assert the raw computed ratio separately so the test fails loudly if the formula drifts.
case 1 is the harder one though. once the LLM is the oracle, how do you keep the scenario fixtures from just becoming whatever the model said last? did you pin the expected verdicts by hand after reading each note?
In some cases exact thresholds make sense - for example, shipping a product that meets 100% of CUJs, yes - that’s needed so we know all promised features are met.
But saying code coverage should be 80% and you are at 78%, some random test could get you over 80% but one has no idea if that added any value.
I will use a mix of 2 for expected verdicts - some I do by hand, if a test is too fragile or keeps finding different issues or keeps failing for various reasons, I will look at it closer. I will also bring in a better LLM which is likely expensive and from a different vendor to review. Often I use expensive other vendor LLM for review and document issues but get a cheaper LLM to implement. This works for me usually. I will also make expensive LLM to do a deeper verification based on logs. Several cases deeper inspection on say logs especially where I am using adversarial models, looking at LLM responses I discovered some hidden issue which the shallow verification missed.