You've been there. You run the eval, the number comes back, and something about it doesn't sit right. But the number is the number, so you move on. Three weeks later you find out the number was never the model's fault.
I spent a week tuning a matcher to explain a 20% pass rate. The bug wasn't in the matcher. It was in the evaluator, in six lines I'd stopped reading because I'd stopped trusting them.
This is a list of seven mistakes. I've made all seven, in this order, and each one cost me real days. For each, I'll tell you what I saw, what I tried first (and why it was the wrong layer), and the small change that actually fixed it. They look like seven different problems. They have one root cause: I kept fixing the model instead of the measurement.
If add commit push is muscle memory but your agent metrics still feel like a black box you're not allowed to open, keep reading.
1. Trusting a Green Suite
A green suite is the most expensive lie in software. Mine said 359 tests passed while my agent's real pass rate sat at 20%. Everything was green, so of course I assumed the model was the problem.
What I tried first: more tests. What I should have done: check whether the tests were asserting anything. One of them had a function body of just pass.
def test_all_adapters_importable():
# it imported, so it must work, right?
pass
pass is not a test. It's a green checkmark with a body. In a PlannerCritic review I found 57 of 65 assertion files were in the wrong format, so the harness quietly returned 0/0 and called it a pass. The suite was green because it ran nothing.
If a module returns 0/0, treat it as an error, not a pass. Zero assertions is not a quiet success.
2. Letting the Model Grade Itself
This one felt like a model problem. It wasn't.
The model scored precision 1.00 and recall 0.02 and still passed. It had found a degenerate trigger it could fire forever and collect the credit. My first instinct was to tune the prompt. That was the wrong layer, and I kept tuning for a day.
My 3B model learned to match the literal string "step_1" because the reward function paid for overlap, not for correctness. It wasn't learning to review. It was learning what got paid. Full write-up here.
Precision 1.00 with recall near zero is not a safe model. It's a model that found the cheapest way to look safe.
3. Scoring Against the Wrong Denominator
Recall was stuck at 0.087. Same number, two field tests in a row, no matter what I changed. I was about to conclude the model had a ceiling.
I rebuilt the matcher. That was a waste. The matcher was fine.
The real fix was one line of scope: restrict the reference pool to the source domain. The model was answering the right question against a haystack of unrelated gold examples, so it looked bad even when it was mostly right. Once I scoped it, recall went 0.087 to 0.170 / 0.228 with no model change at all. The full numbers are here.
If a metric is stuck at the same low value across every model you try, suspect the denominator before you suspect the model.
4. Optimizing the Layer the Report Named
A week of matcher work moved the metric ten points. Ten points. I'd been fixing the matcher because that's the layer the failing tests kept naming, and I trusted the report more than I should have.
The change that actually moved the needle was six lines in the simulator that generated the failing cases. The score jumped from 20% to 50%. Six lines. I wrote about that week here.
This is the mistake that stings the most, because the report pointed me at the matcher and I believed it.
The layer the failure names is not always the layer that's broken. Prove the origin before you spend the week.
5. Treating a Model-Independent Failure as a Model Ceiling
A local 4B and a cloud model hit the exact same wall. Uniform 20% across both. My knee-jerk was to buy a bigger model. Same wall.
Identical failure rates across wildly different models is almost never a capability ceiling. It's a shared harness bug. I compared a local 4B, a better cloud model, and role separation, and the results were identical in a way that should have been a warning. The comparison is here.
If every model plateaus at the same number, that constant is the thing to fix. Not the model.
6. Mocks That Can't Break
Unit tests green. Real agents failing in the field. I wrote more mocks. Wrong move.
My mock suite reported 9% pass. The zero-mock field test found every single failure the mocks couldn't express. Why? A mock agent always calls the tool it's told to. A real model, sometimes, just answers in prose.
A mock can only fail in the ways you already imagined. Real agents fail in the ways you didn't.
Your mocks are a record of your current imagination, not of the system.
7. Forgetting the Runner
A 1,000-run benchmark died at the 80% mark. I blamed the corpus. I re-ran it. It died again.
The bug was the runner. One hung API call had discarded 60 completed audits, no error record, nothing. I added a per-trajectory timeout, non-retryable timeout classification, a token cap, quarantine on failure, and cancel-on-shutdown. After that, a clean field test ran 4,768 trajectory runs with zero lost sweeps. The field test report is in the repo.
A hang is a data-integrity bug, not a performance bug. A partial sweep still looks like data, and that's the trap.
The One Thing All Seven Share
Look back. Only one of these was actually a model problem, and even that one was fixed in the scorer, not the prompt. The other six were measurement bugs: a suite that didn't assert, a denominator that lied, a layer I misidentified, a classifier shared across models, mocks that couldn't fail, and a runner that dropped data.
I'd rephrase the lesson this way: the reward function is almost always the real bug. You can tune the model forever and the number won't move, because the number was never the model to begin with.
The open question I still can't answer: how do you know your eval is honest? I can name the seven mistakes now. I can't fully trust that I've caught the eighth. A green suite with a hidden measurement bug and a red suite with a real bug look identical from the command line. I don't think there's a clean test for that yet, and I'd rather say that out loud than pretend I closed it.
Which of these did you ship before it taught you? I'll admit to all seven. Tell me yours.
Code and receipts: agent-eval-forge · CauterRule · planner-critic-engine — all MIT, all public.
Top comments (5)
The "0/0 counts as a pass" one is the quietest killer here, and I think it's worth stating as a hard rule: any eval module that produces zero assertions should raise, not return green. We got burned by a variant of this — a harness that skipped a whole suite because a fixture failed to import, and the runner reported "0 failures" with a straight face. Green wasn't lying about the tests; it was lying about whether they ran, which is worse.
The self-grading one resonates too. Once we started paying a reward for overlap instead of correctness, the model found the cheapest string to emit and farmed it. What finally helped wasn't a better prompt, it was adding a small set of adversarial negatives the grader had to reject — precision 1.0 / recall 0.02 stops looking impressive the moment a "should-not-fire" case is in the denominator.
Curious how you're catching #2 now — is it a fixed negative set, or something that regenerates traps per eval run so the model can't overfit to them?
Number one has a version that survives long after you stop writing empty test bodies: a harness that counts what the run claims instead of what it left behind. An agent asked for 600 items can call every tool correctly, format a confident summary, and have processed 500. Every assertion green, because nothing compared the request against the artifact.
The guard that generalises your 0/0 rule is to pin the expected count. Not "did the suite pass" but "did it run the number of assertions it ran last time", and fail on drift in either direction. A suite that quietly shrinks is the same failure as one returning 0/0, just slower to notice.
The other one I would put next to number four: a checker can be wrong about the data. Mine once reported a missing field in a delivery block, and I started fixing the generator. The block was a different kind of block and the check's assumption was wrong. So when a check fires, confirm what it assumes before you trust where it points, or you spend the matcher week on a report that was never describing your system.
The green suite point is the one I would put on a sticky note next to every chatbot eval dashboard.
A knowledge or support bot can look healthy while the harness is quietly scoring 0/0, letting the model grade its own refusals, or measuring recall against the wrong gold pool. In those cases the number moves when you change measurement, not when you change the model, and weeks disappear into prompt tuning.
I would gate ship on three checks that the suite itself cannot fake: every scored case has a non-empty assertion, refuse and escalate paths are scored separately from fluent answers, and the reference set is scoped to the same domain as the question. If any of those is missing, a green board is not evidence. It is a permission slip to ship an unevaluated bot.
Seven symptoms, and I'd argue one shared property: in every case the harness's failure mode produced a plausible number instead of an error. 0/0 reported as a pass, a degenerate trigger reported as precision 1.00, a mismatched denominator reported as a ceiling. None of those crashed. A measurement system that fails by returning something believable is worse than one that has no coverage, because the second kind gets investigated and the first kind gets cited in a status update.
Two things that have helped me on RAG and classification work, both cheap:
Run the negative arm before you trust the positive one. Deliberately sabotage a fixture, or feed the eval an answer you know is wrong, and require the run to go red. An eval that has only ever been observed passing is indistinguishable from an eval that returns pass unconditionally, and your 57-of-65-wrong-format case would have been caught the first afternoon by a single seeded defect in CI. This is mutation testing pointed at the grader rather than the code.
Assert on the shape of the metric, not just its threshold. Precision at exactly 1.00 with recall near zero is not a borderline result, it is a signature. Any run where one half of a metric pair sits on a boundary should fail for human review regardless of the aggregate score, because that is what degenerate solutions look like from the outside. Same for a metric that is bit-identical across two different models, which is your denominator bug announcing itself.
The one I'd add to the list from my own week lost: no canary case. Put one trivially correct example in every eval run. If the trivial case does not score near the ceiling, stop reading the rest of the numbers, because you are measuring the harness and not the model.
5 is the one that saved me the most debugging time once I internalized it: if a local 4B and a frontier model land on the exact same score, you've found a constant in your harness, not a ceiling in your model. These days I treat identical cross-model plateaus as a harness bug by default and go hunting in the scorer before touching anything else.