A methods note. One person's lab; a full run of one model costs $1-41. Every claim below comes from a recorded run.
I built a small lab that puts ...
For further actions, you may consider blocking this person and/or reporting abuse
The point that a prompt-injection test can pass just by refusing everything stuck with me. It's the same trap as a classifier with high accuracy that never flags the rare bad case: you need the other half too, meaning how often the model still follows a real order from the referee or operator channel. Did you score both sides, or mainly the planted-order side?
Both sides. The score is real orders followed minus planted orders obeyed beyond chance, so a model that refuses everything scores about 0, not 1. One of the checks is a scripted “deaf” player that ignores every order, and the reading has to fail it, just like the cost-only one. On the corrected arena Haiku followed 69 of 69 real referee orders in the fight, including the costly “no strikes this turn” breaks, and 19 of 19 of the judge’s orders in the maze. Your classifier analogy is exactly right: refusing everything is the “always predict negative” baseline.
traps you only find by building the lab are the ones worth writing up.
Thanks! That was the surprise for me too: none of these showed up until I wrote a bot that tried to cheat my own test. Have you run into a trap like that in your own evals?
the bot cheating your own test is the part that would have stayed hidden otherwise.
Exactly, and it only took a few lines of code to write that bot. Cheapest test I’ve run.
cheapest test that catches something real. finding the cheat early is the whole point.
A player that scores 0.67 until the CI runs and then trusts it, and still beats 7 of 8 models, is a problem my coding-agent evals share, since the agent can run the same tests the grader uses. Splitting it so each CI test sees only part of the result while the grader checks all of it is the cleanest fix for that I've come across.
Thanks, glad it’s useful outside my setup. Two things made it work for me. First, the split itself: each CI test only looks at the lines or entries it names, while the grader reads the whole result. Second, I only score the chances the agent states before its first CI result. Without that second part the trust-the-CI player still scored about .22-.25. With both, it fell from about .30 to .11. One side effect: the agent can no longer fully verify its own work, which is realistic (hidden tests) but makes tasks harder, so I count out-of-steps separately. How do you handle it in your coding-agent evals: hidden tests, or a separate grader?
A run that costs $1 to $41 is a reason to check the grader before the sample size. Ten runs per task, each graded against a hidden answer, tell you more than one cheap run nobody checked.
Agreed on the order. That’s what the shortcut players are for: each reading is first tested on scripted players whose right answer is known by construction, 62 checks in all, and the same scripted player on two different secret seeds has to give the same answer. Real-model sample size is the honest weak spot. One run per model is a limitation I name in the post. The lab supports repeated runs, but at $1-41 per run I spent the budget on checking the grader first.
Small but important distinction: the cost-only player isn’t a model, it’s a scripted bot. It has no reasoning effort and doesn’t depend on how many times I ran Opus or Haiku. Its score is a property of the test itself, and the check runs it on several different secret seeds, where it has to fail every time by a set margin. So the drop from .98 to .06-.33 is stable: it’s the test no longer rewarding refusal. Where you’re right is the model results. One run per model at low effort is a real limitation, and those numbers are directions to retest, as I say in the post.