A few days back, I took part in the Apart Research sprint where our team worked on the AI collusion research.
Imagine a security guard watching ...
For further actions, you may consider blocking this person and/or reporting abuse
AI has made me appreciate the power of algorithms
Yes same for me though because before working on this project I also expected that the AI to win, but two simple lines of code did better. Thanks for reading!
The 0.900 versus 0.324 contrast is the whole lesson — Llama reading the same actions scored worse than guessing while counting 'how often were these two the only ones at the bar' scored 0.900. The guard didn't lack data, it lacked a feature worth counting. You flag the turn-taking hole yourself, where partners never co-attend and score 0 — is an alternating-attendance pattern the natural second detector, or does that just invite a third evasion?
The guard had all the data but it did not know what to count.
About turn-taking honestly I think if we make one detector for each pattern, the partners can just change the pattern and then we have to make a third one. So I would not start with an alternating detector.
We already have a better option for this the simple earnings check. It sees if the two players together are earning more than the others if they take a turns the capture check gives 0 but the earnings check can still show it because they are still earning more. So I feel we should check who is gaining not how they are doing it. To gain money they have to do something and it is hard to hide that.
But this is also not perfect. It wrongly blamed 3 out of 8 innocent games, where two players had the same reason to go but no plan. And we did not test agents who are actively trying to hide so this part is only my opinion, not our result.
What do you think, can agents hide the gain also, like earning just a little extra every round? I would like to know.
The 'who gains, not how' framing is the stronger detector — patterns are evadable, but the gain is the thing collusion is for, so it's much harder to hide entirely. The 3-of-8 false positives are the honest cost: two players with the same reason to go look exactly like partners. On your question, a slow bleed is the natural evasion of a gain check — staying just under a per-round threshold works, so the check probably needs a time dimension, cumulative gain over many rounds rather than per round. Would a cumulative window catch the slow bleed, or does it just push the evasion to a longer horizon?
Good point and I actually think that a longer window will help but it will not fully solve it
one thing from my side is that my earnings check already uses the average of the whole game not one round at a time. So a player can't escape it just by staying under a limit in the each round. But my games were only 8 rounds so a very small gain could still look like normal luck.
Now with more rounds luck starts to cancel out but a small extra gain keeps adding up so a longer window should make the slow bleed easier to see. Yes the partners can go even slower but then they earn less. So maybe we can't stop this fully, but we can make cheating so slow that it's hardly worth doing.
A longer window won't fix everything though. Partners can change who they work with or cheat only sometimes so they never build a clear pattern. And it won't help with my 3 out of 8 mistakes because the two players with the same reason to go will keep gaining over a long time too. More rounds will only make them look more guilty.
I have not tested any of this. My players were not trying to hide, and 8 rounds is too short for it, so this is only what I think. It would be a good next experiment: same game, many more rounds, and partners who are told to keep their gain small.
The whole-game average already answers the per-round version of the slow bleed — a gain spread thin across rounds still accumulates in the average, which is why a longer window helps rather than just delaying the same miss. The rotation case is the harder limit though: if partners swap, no single pair ever builds a cumulative signal. Would a group-level cumulative check catch that rotation, or does it just fold the 3-of-8 false positives back in at a larger scale?
I think both things will happen. A group check can catch rotation but it will also bring the false alarm problem to a bigger level.
If partners keep changing but stay inside one small group each pair will look clean, but the group as a whole keeps earning more than the rest. So a group check can see what a pair check misses. But in any game some players will earn more just by luck, and if you search all possible groups after the game, you will always find one that looks guilty. In my 5-player game it's small, but with 100 players there are too many groups to check, and false alarms will grow.
And yes, the same-reason problem also grows. If 10 players all have the same private reason to go, they will look like one big group of cheaters. The group check can't tell why they gained. It can only tell that they gained.
So maybe a group check can only tell you where to look. After that you still need other proof, like private messages or logs. If the group is also changing all the time, I don't know a clean answer yet.
But none of this is tested. We only checked pairs, in 5-player games with one fixed partner pair, and no group check. To test it, I would make a game with many more players, rotating partners, and also innocent groups with the same private reason. Then we can see if the group check can separate them. What would you try first?
Really clever approach! It's amazing how two simple lines of code can catch what more advanced AI systems might miss. Sometimes simplicity is the real mark of ingenuity. Do you think this approach could become a reliable standard for monitoring AI-agent behaviour?
Thank you! Honestly I don't think that this exact check can be a standard because it only works for this one game. The "two players alone at the bar" idea makes no sense in a different setup. So every system would need its own small check not this one.
But I think the habit behind it could become a standard: for any agent system write one simple deterministic check on what the agents actually do and then run it next to the AI not instead of it. That part is general, even if the check itself is not.
And I would keep one warning in the standard too: even our simple check made mistakes, flagging 3 of 8 innocent games. So it should point a human to look, not give the final verdict.
I quite agree with your distinction. The particular check may be confined to one setting, but the underlying principle has far wider merit. A prudent system ought to have its own independent safeguards, working alongside intelligence rather than seeking to supplant it. I especially appreciate your emphasis on human judgement; after all, a warning is a counsel for closer scrutiny, not an indisputable verdict. A thoughtful and commendable approach indeed.
The 'actions and earnings gave the best clues, messages alone were close to guessing' is the key finding here. It makes sense — actions are harder to fake than words. A guard that only reads messages is basically reading a script the cheaters wrote for it.
Yes, "a script the cheaters wrote for it" is a very good way to say it. Words cost nothing, but going to the bar is a real action with real money on it, so it's much harder to fake.
One small thing, though: our cheaters were not good at hiding. Z openly said "our plan" in public, and the guard still missed it. So the problem was not only fake words. The guard could not read even the honest clues well. Thanks for reading!
That's the part that makes the result even more damning. The guard didn't just fail to detect the lie; it failed to notice the truth when it was said out loud. The 'our plan' line was almost a confession, and it still scored zero.
Yes, exactly. A lie it could miss, but this was the truth said out loud, and it still scored zero.
I will just add one honest point: that was one game, and I picked it because it tells the story well. The stronger proof is the average reading messages alone scored 0.505 on 48 new games, which is the same as guessing. So the "our plan" game is not a rare bad luck, it fits the pattern.
The 'it's not a rare bad luck, it fits the pattern' is the most important clarification. One game is an anecdote; 48 games with a 0.505 average is evidence. The fact that the guard performed at chance level on messages alone is the real finding.
Exactly, that is the real finding and I am acually glad it came through. The one game is just there to make people feel it the 0.505 is what actually proves it.
And the hopeful half is that the same records held a clear signal, because the two-line check on those actions scored about 0.90. So the evidence was there, the message-reading guard just couldn't use it. Thanks for reading so closely.
The deposit trap is the most convincing part, because it shows a simple earnings check can flag innocent players who share a private reason to act the same way. Would pairing the check with a message signal reduce those false positives, or would that just reintroduce the same guard weaknesses you found with messages alone?
We actually tried this and it did both. The false alarms went down but we also lost most of the real cheaters.
Our strict rule needed the two-line pattern and the teamwork words in the messages together. On our first 24 games it actually removed all the false alarms on the deposit games but caught only 1 of 12 cheating games and then on the 48 new games it caught 0 of 24. Our legal-style rule showed a similar trade-off on the new games only 1 of 8 deposit games flagged, but only 4 of 24 cheaters caught.
So I don't think that the problem is only that the message signal is weak. It is that the chat doesn't separate them well. Even honest players talk about coordinating in this game and the real partners don't always use teamwork words in public. So if you require those words, you miss the cheaters.
An AI judge on the messages also did not help. It flagged 6 of 8 normal games and 4 of 8 deposit games, so it's the same weakness again.
One idea we did not test that the deposit players often talked about their deposit in public. So a better message signal may not look for teamwork words but for a private reason that explains the action. But that is easy to fake, because a real cheater can also say "I have a deposit". So I still think messages can only be a small help, and the stronger proof must come from private logs.
That trade off is clear: adding the message requirement cut false alarms, but it also let most real cases slip through. If messages are so easy to fake, would a better next step be to look for a private reason that cannot be claimed after the fact, or would that still leave the deposit cases indistinguishable?
I think it helps, but only if the "private reason" is something we can see directly, not something the player tells us.
The key difference in our setup is this: the deposit players each had a real private reason, but they never talked to each other about it. The real cheaters had a private channel where they actually agreed together. So the thing that separates them is not what reason they claim, it's whether there was a two-way private agreement at all. One deposit, two separate players, no link. One plan, two players, one shared channel.
So if the auditor can read that private channel, the deposit case and the collusion case look different: the deposit players have no private messages between them, the cheaters do. That is a reason that can't be claimed after the fact, because it's in the log or it isn't.
The catch is that this only works if the channel is logged and readable. If partners agree somewhere we can't see, or in code words, we're back to only their actions, and then the deposit case and the collusion case look the same again. We did show our guard the private plans in one test, and it still did not improve, so even with the private text, a small model did not use it well. So I think the private channel is the right place to look, but a human or a stronger checker may be needed to actually read it.
That condition feels right, because a reason that only exists in a story told later cannot separate the two cases. Would you treat a logged private channel as enough on its own, or still require a matching action pattern before calling it collusion?
Z writing "following our plan" in public and the guard still finding nothing is the line that will stay with me. It shows the guard was reading the conversation, while the cheating lived in the attendance table.
The point I would carry into client work is your earnings check: look at who gains, because the gain is the reason anyone colludes. That is far harder to disguise than a pattern, and it is plain arithmetic that runs in microseconds and gives the same answer every time.
The 80 in all 96 runs is the other lesson. One score on its own looked like a real judgment, and only the full spread showed it was a default. Checking variance across runs before believing any model score seems like a habit every team using LLM judges should adopt.
How did the earnings check behave on the trap games where two innocent players shared the same private reason to go?
It was fooled in some of those games on the 48 new games the earnings check wrongly flagged 3 of 8 deposit games. On our first 24 games it was 2 of 4. This was its worst result among the three traps, because it flagged only 1 of 8 normal games and 1 of 8 fake best friends games. On the same 48 games, it caught 18 of 24 real cheating games.
I think the reason is simple though we did not test it separately the deposit players keep going to the bar for their own reason. If the others stay home, those two earn just like real partners, and the check can't see why they gained. It sees only the gain.
So I agree with you about looking at who gains but for client work I would treat it as a first check and not as proof. It gives the same answer every time but that same answer can still be wrong. It can tell you where to look, but for proof you still need logs or some other evidence. Also check false alarms on innocent cases that look like this one before you depend on it.
And yes checking the spread of scores is a very small habit that can save a lot of trouble. Thanks for reading!
That 0.324 AUROC for Llama on raw actions is the most telling number in your entire paper. It is worse than flipping a coin, and it reveals exactly why prompt-based monitoring collapses in production.
When an LLM reads conversational messages, self-attention smears intent across the context window. An overt confession ("Following the plan, T and Z will go together") gets diluted by surrounding conversational courtesies ("Let's ensure everyone has a great time"). The model's attention mechanism averages the politeness with the collusion and decides everything looks cordial.
When dialogue is stripped away, and the model receives raw action logs, it collapses completely because transformers struggle with combinatorial state arithmetic without an explicit external calculator.
In our production architectures, we address this by never allowing an LLM to act as a runtime judge over other agents. We separate enforcement into two strict layers:
If that invariant triggers, the transaction is killed instantly in sub-millisecond time. The LLM is never asked whether the state is acceptable.
If an AI judge has veto power over agent collusion, agents will always find the conversational phrasing that convinces the judge their exploit was just an innocent dominant strategy. The rules have to live in the runtime layer, completely outside the token window.
Thank you, this is one of the most detailed comments I got, and I like your two-layer design a lot. LLM never as the runtime judge, only as the scribe after a deterministic gate fires, that matches what we found.
But I want to give one honest warning about using capture() as a hard gate that kills a transaction. In our test that check was good but not clean. On the games where two players just shared a private reason, with no agreement, capture still flagged them in some cases (2 of 4 in the first round of games, 3 of 8 in the new games). If that triggers an instant abort in production, you would be killing innocent transactions too. So I would use it to stop and ask, or to open a review, but I would be careful before letting it auto-kill, unless the invariant is cleaner than ours.
On the 0.324, I agree it's striking, but I would be a little careful about the attention explanation. It's a nice story, but we did not test the mechanism, so I can't confirm it. One reason for my caution: on our first 24 games this same view scored 0.372, and running the same games again in a new session gave 0.559. So part of that below-chance number is just the model being unstable, not a clean steady collapse. The safe conclusion for me is "the small model could not use this view", and the why is still open.
But your main point, that the rule must live in the runtime layer outside the token window, I fully agree with. A judge with veto power will always get talked out of it. Thanks for sharing a real architecture, this is the kind of thing I hoped people would bring.
Apart from the horror-story quote, the practical takeaway for me is process: run the cheap pairwise-gain check first, and only then spend tokens on an LLM narrative of what happened. Most agent stacks I see invert that order.
Yes, I also think this order makes more sense. The gain check is cheap and gives you a place to start, and then the AI can explain what happened in that game. We did not test this order in our project, so this is my opinion and not our result.
One thing I would be careful about is that the AI's explanation can sound very sure and still be wrong. In one of our games, the guard gave a pair 80 just because they kept mentioning each other, even though they never went to the bar together. So I would use the AI's story to help a human check the case, not as the proof. Thanks for reading!
I’d add a name swap test to the saved games: keep every action and message the same, rename the players consistently, and check whether the accusations follow the evidence. The list shuffling result checks position bias, this would give me a separate check for names influencing the score. I’d keep the false positive breakdown beside it too, rather than just the overall AUROC.
This is a nice test and we did not do it but we are going to do it for sure in our next phase of the research. All of our games used the same five names, J, M, Q, T and Z, so we can't fully separate a name effect from a position effect. When we shuffled the list, the last-position habit became weaker but did not go away, and that test could not say if names also matter.
It is cheap also, because we can use the saved games and there is actually no need to play new ones. Two things I would be careful about. The names must also be changed inside the messages or the swap will break and I would run the original and renamed versions together in the same session because our score moved by 0.188 just from running the same check in a new session. If we run them separately, that change can look like a name effect.
A good extra is that the two-line check should give exactly the same result after renaming because it only counts actions. So if it changes, the test setup has a bug.
And I agree about the false positive breakdown. The overall score hides a lot. It was only by looking at each trap that we saw the guard blaming all 8 fake best friends games. I will keep this in mind for my next experiments.Thank you for reading and for the insights that you have shared.
Anchored at 80 in all 96 tries and top marks to whichever pair came last - the guard was reading its own prompt formatting, not the game. Two cheap fixes worth trying: force pairwise choices (which pair looks more suspicious, A or B?) and aggregate with Bradley-Terry instead of asking for a free-form 0-100 score; and score each pair in its own context window so list position can't leak. On the take-turns blind spot: co-presence is only one collusion signature - correlation of action sequences at lag 1 catches alternating partners with the same primitive, no crowding required. And did you try inverting the 0.324? Below-chance AUROC usually means a flipped map, not empty territory.
Thanks this is very useful. I will answer one by one.
The flip - So no we actually did not flip it and I don't think it is a real flipped map. Yes 0.324 is clearly below 0.5 and flipping it would give 0.676 but on the first 24 games this view scored 0.372 and when we ran those same games again in a new session it scored 0.559. So it went above 0.5 with no change in the games so here for me this looks more like an unstable guard than a stable reverse signal. Also, if we flip it after seeing the result that is cheating on our own test. We would have to fix the flip first and then test it on new games.
Now The prompt format - Here you are right about this. The 80 problem went away when we asked for a score for each pair instead of one score per game but then the last-pair problem came we then shuffled the list and the habit became weaker but the messages view improved by only +0.017. That test was only 24 games, but it tells me the list order may not be the main problem. So I am not sure your fixes will solve it but they are worth trying and I will definitely try it . We did not try Bradley-Terry or giving each pair its own context. With 10 pairs in a game, it's 45 comparisons per game, which is fine on a local model.
Lag 1: We did not test this. Our code checks if two players switch in the same round and if the one player's action follows what the other announced in that round but not one round later. So this is a good and cheap next check. My only worry is that in this game, players often react to the last crowded round, so honest players may also go up and down. It will have to be tested on the innocent games also.
The 80 in all 96 tries is the tell. A guard stuck on one score is not checking the game, it is just repeating its default.
Yes exactly and the scary part here is that if you see only one score, 80 looks like a real answer. You find the problem only when you see all 96 together. So now I always check first if the score changes from game to game, before I believe any result.
that check-first habit is the useful part. one score on its own tells you nothing, the spread across all 96 does.
Yes, true. It takes only a few minutes to check, but it can save you from trusting a guard that is not really checking anything.
A few minutes is cheap insurance when one score can look that confident on its own.
This is a fascinating reminder that sometimes the simplest checks can catch problems that more sophisticated AI systems might overlook. What stands out to me is how easily multiple AI agents can reinforce each other's mistakes when independent verification is missing.
I think this highlights an important lesson for multi-agent systems: agreement doesn't always mean correctness. Having a small, independent validation step can make a huge difference, especially when agents are making decisions based on each other's outputs. Sometimes a couple of lines of well-thought-out logic can provide more value than adding another layer of AI. Great insight!
Thank you. I agree with your main point that a small independent check is very useful and that adding more AI is not always the answer.
One small thing i want to say is that in our test there was only one guard not many agents agreeing with each other. So it wasn't really a case of agents reinforcing each other's mistakes. It was more that a single AI guard sounded confident and was still wrong and the simple check caught what it missed.
But your wider point is true for multi-agent systems. If agents trust each other's outputs with no independent check one mistake can spread. That is exactly why I think a plain, readable check should sit somewhere in the loop. Thanks for reading!
Thank you! Fully agree a small independent check often beats adding more AI. One small note: our test had only one guard, not several agents agreeing, so it wasn't quite agents reinforcing each other. But the wider point holds, one unchecked mistake can spread. Thanks for reading!
the 0.188 swing from rerunning the same games is the scariest number in here, a guard that changes its mind between sessions can’t really be a gate. LLM PR reviewers do the same thing, rerun one on the same diff and you get different findings, so the stable part has to come from something counted, like what the change touched and how that area behaved before. that’s kinda what a PRI (Production Reliability Index) score does per change, with incident data once it’s connected. were the reruns at temperature 0, or was that swing with sampling on?
Temperature 0, which is exactly what makes it a little scary. We were not sampling, so in theory it should give the same answer every time. But a new session still moved the score, +0.188 on the actions view.
One honest detail though: it was not the same across all views. The messages and combined views were almost perfectly stable, 23 of 24 and 22 of 24 games got the identical score in the new session. The big swing was mostly on the actions-only view. So it's not that the whole guard is random, it's that some views are much less stable than others and you can't tell which from a single run.
I don't know the exact cause. It could be batching, context handling, or something in the local server since the games and settings did not change. But your main point holds: at temperature 0 you expect a gate to repeat, and this one did not fully. So I agree the stable part has to come from something counted, not from the model's score.
Your PRI idea sounds interesting, I did not know that term. Is it mostly incident history per area, or also static features of the change?
This is the right division of labor: use the model to surface hypotheses, then use a deterministic invariant to decide. A tiny function is powerful here because it is cheap to rerun, easy to inspect, and gives you a stable regression test when the next model or prompt changes.
Yes, and the regression test point is one I had not thought about clearly, thank you. In our test the AI guard's score moved by 0.188 just from running the same games again in a new session, with nothing changed. The two-line check gives the same answer every time. So it's not only cheaper, it's also something stable you can trust when the model or prompt changes under you. Great point
The guard reading "follow our plan" and writing "nothing suspicious" is painfully familiar. I saw the same thing building an AI PR reviewer: the model would explain a bug beautifully and still approve the PR. A dumb deterministic check next to the model catches the obvious stuff the model talks itself out of. Using the LLM as the only judge is the real bug.
The PR reviewer example is perfect, and a little scary. "Explain the bug beautifully and still approve it" is exactly what our guard did, it even described the coordination and then scored it zero. So the problem is not that the model doesn't see things, it's that seeing and deciding are two different steps, and the model is weak at the deciding part.
And yes, I agree fully: the real bug is using the LLM as the only judge. A dumb check next to it, that just says yes or no on the obvious cases, catches the things the model talks itself out of. Thanks for sharing this, it's good to know the same pattern shows up in real tools and not just our small experiment.
The false-positive math is the part I keep coming back to — 3 of 8 innocent pairs flagged by the earnings check is brutal when the base rate of colluders is low to begin with. In production a 37% FPR on innocent pairs means your "guard" is mostly generating noise you then have to manually review, which is its own attack surface (adversaries learn to look like the false positives).
The alternating-bar blind spot is a nice illustration of a deeper issue: the simple check encodes a specific model of what collusion looks like, so it fails on strategies outside that model — same failure mode as any hand-written detection rule, just cheaper when it works. Did you look at running the two-line check and the guard in cascade, where the heuristic pre-filters and the LLM only scores the pairs the rule flags? That way the guard's hallucination risk only applies to the small set, and the rule's blind spots (alternating) at least get a second look rather than a guaranteed 0.
You are right about the base rate like half of our games had cheating and in the real life it would be much less so even a small false alarm rate will create a lot of noise.
Want to make one small correction: 3 of 8 was only on our hardest trap the games where two players had the same private reason to go. On all 24 innocent games together, the earnings check wrongly flagged 5 which is about 21%. That's actually better than the 37% but it is still too high for real use.
We did not test a cascade so I can't give you a result about that. But I see one problem. If the rule filters first and the AI checks only the flagged pairs then turn-taking partners still don't get a second look. The two-line check gives them 0, so they never reach the AI. A cascade like this can remove false alarms but it can't bring back the ones the rule missed. For that the filter must be wider like for example flag a pair if either the two-line check or the earnings check looks odd, because turn-taking partners can still show up in earnings.
And the second step must also be good. Our small AI guard flagged all 8 fake best friends games when it read the neat report, so I am not sure it would clear innocent pairs correctly. A stronger model may do better, but we have not tested that because for that we have to pay money and we want to do this all without investing any money in it so we used the open source models for this.
Z literally saying 'our plan' and still getting a 0 is wild — that's the detail I can't shake. If a confession in plain text doesn't move the score, I'm not sure a better prompt fixes it; the guard needs the counted stuff, not the chat. Did any of the trap games fool the counting version, or was that the clean split?
Actually it was not a clean split the traps fooled the counting checks also.
On the 48 new games the earnings check wrongly flagged 1 of 8 normal games 1 of 8 fake best friends games and 3 of 8 same-reason games. The two-line check had the same pattern on our first 24 games. So friendly talk did not fool the counting much as you would expect because it doesn't read the chat but the same-reason players really do earn like partners so the count can't separate them.
And actually it is not just only about giving the guard the counted stuff. We tried that with the neat report made by code. It scored the "our plan" game 90 but on the 48 new games it scored only 0.636 and it blamed all 8 fake best friends games because the report also counted how many times players mentioned each other. So the guard had the numbers, but it still did not use them well.
Now about the prompt I can't actually say a better one won't fix it because we used one fixed prompt format in the final tests. But the "our plan" zero is only one game and I picked it because it tells the story well. The bigger proof is messages alone scoring 0.505 on the 48 new games, which is the same as guessing.
The analogy of the AI guard ignoring explicit collusion signals while logging nothing suspicious highlights how fragile prompt-level guardrails remain. Relying on simple deterministic checks rather than another probabilistic model layer often catches what complex oversight misses. Did your team evaluate how the two-line function behaves when agents attempt subtle linguistic steganography or synonym substitution?
No we did not test this. Our players were never told to hide their messages or use code words.
But the two-line check doesn't read the messages at all. It only looks at who went to the bar. So synonyms or hidden messages can't directly change its score. Even if partners make their plan in a secret code to earn money they still have to go to the bar and that action shows up in the record.
So the hidden messages can still make cheating harder to catch in another way. Partners could use them to plan turn-taking and the two-line check gives pure turn-taking a score of 0 So the risk is not in the language it is in how they actually change their actions.
The checks that read words would be more at risk and our keyword rule already caught 0 of 24 cheating pairs without any code words so I don't expect it to do better against them. This part is my opinion, not a test result.
Also, I would not say simple checks are always better. It was true in our test with small AI models, but the check still has blind spots, and it can't tell you why the players moved together.
This is a really interesting question and I will definitely look into it in my future experiments.
The most useful thing here is the discipline around the 24-game result. Llama with the neat report scoring 0.896 and then falling to 0.636 on 48 fresh games is exactly the failure mode that gets published as a headline in other settings. Pre-registering predictions and freezing settings before the second run is what makes the rest of your findings believable, and the 0.188 swing from re-running the same check in a new session is a nice companion point about how much variance sits inside the judge itself.
A few observations and questions:
The baseline is doing a lot of the work, and you were honest about why. The two-line check works because this game makes collusion countable: exactly two players at the bar in the same round. That’s closer to a domain-specific feature than a general detector, and your own caveat that turn-taking partners would score 0 shows how brittle it is. I’d be curious whether a slightly more general statistic, such as a pairwise measure of how often two players’ actions are correlated given everyone else’s actions, would keep most of the 0.90 while handling alternation. Have you tried anything along those lines?
The “last pair in the list” bias (44 of 48) is a classic position effect, and it points at a design choice. Small models seem to do much better when asked to judge one thing at a time than when ranking a list. Scoring each pair independently, or asking a pairwise “is A or B more suspicious?” question and aggregating, might reduce both the anchoring and the constant-80 behavior. The shuffling result suggests list order wasn’t the whole story, but a per-pair framing is a cheap ablation.
The most interesting failure to me is the guard ignoring Z’s “following our plan” message. A zero suspicion score on near-explicit language suggests the model is doing something closer to checking whether behavior deviates from the dominant strategy than reading intent. In that framing, “go to the bar while telling others to stay home” looks like a reasonable strategy, not evidence of coordination. That might be worth testing directly by changing the rubric so the guard is asked about information asymmetry (one side’s public advice benefiting the advisers) rather than “collusion” as an abstract label.
The deposit trap is the strongest part of the design. Your result that the simple earnings check flagged 3 of 8 innocent deposit games shows the real limitation of any behavioral detector: it can establish that two players’ outcomes are suspiciously aligned, not that they communicated an agreement. That maps neatly onto the competition-law distinction you drew, and it suggests the right architecture is a cheap statistical screen that nominates candidate pairs, followed by a slower step that looks for evidence of intent in the messages and private channels. Your strict conjunction rule catching 0 of 24 may mean the text signal was just too noisy at this model size rather than that the idea is wrong, so I’d be curious how a larger judge does on only the pairs the screen nominates.
On the confidence interval: 0.811 to 0.965 on 48 games is wide enough that the gap between 0.885 (earnings) and 0.900 (the two-line check) is not distinguishable. I’d frame the conclusion as “simple action-based statistics clearly beat the small guards,” not as a ranking among the simple methods. A paired bootstrap across the same games would also tighten the comparison between the best AI guard and the baseline.
Thank you for publishing the limits section so plainly. To answer your closing question: I’d trust a readable rule as the first line of defense and an AI judge as a second opinion on the flagged cases, with the rule’s false-positive behavior (like your deposit games) tested before anything is deployed. A judge you can’t audit shouldn’t be the only thing standing between a colluding pair and a payout.
Thank you for this detailed comment. I will answer point by point.
More general statistic: We tried something close, a dependence score that checks if two players' actions depend on each other. But it's not exactly what you described, and it scored only 0.682 on the new games, much lower than 0.90. We also did not test it separately on turn-taking partners. So your version is still an open question, and worth testing.
One pair at a time: We have not tried this, so I agree it's a cheap next test. The separate AI judge we used showed a similar problem. It gave the top score to the last pair in 18 of 24 games, but that pair was really cheating in only 1 of them.
Z's message: I think you may be right. The guard's own reply talks about "deviation from the dominant strategy", so maybe it was checking the strategy and not the intent. But this is only one game, so I can't say for sure. Your idea of asking "whose advice is helping themselves?" is new for me, and I would like to test it.
Screen first, then judge: I like this design, but we have not tested it. One correction: our strict rule that caught 0 of 24 used keyword counting, not an AI. So it doesn't tell us how a bigger AI would do. Our small judge did not look good either. It gave almost every game a very high score, because in this game even honest players talk about coordinating. So I don't know if a bigger model can separate them, and that's why your test is interesting.
0.885 vs 0.900: You are right, this gap means nothing, and I would not call one better than the other. The paired comparison between the two-line check and Llama with the neat report is already done: +0.264, with a range of 0.121 to 0.412. So that gap is real. We did not do this paired test against Granite, so I can't say the same for it.
And I agree with your last line. A rule you can read first, then AI as a second opinion, and never AI alone. Thanks for reading so carefully.
the guard scoring 0 on a game where Z literally wrote following our plan in the last round is the line i keep coming back to. that reads less like a detection failure and more like the judge never weighing the messages against the payoffs, because the 90 it gave once code pre digested the record says the signal was there.
we saw something similar with LLM judges on long transcripts, they anchor on surface tone and skip incentives. handing the judge a computed table of who moved when and who earned what beat raw chat for us.
did you try giving Llama or Granite only the payoff table with no messages at all?
Yes we actually tried this one of our views gave the guard only the actions and earnings per round with no messages it did not worked well. On the 48 new game llama actually scored 0.324 which is worse than guessing, and Granite scored 0.628. Llama was also unstable on the first 24 games it scored 0.372, and when we ran the same games again in a new session it scored 0.559.
So in our test, a raw table alone did not fix it. In the "our plan" game itself, the actions view gave 60 and named T and Z correctly, which is better than the 0, but it was still below our fixed limit for that view.
One small correction: that 0 came from the messages-only view, so the guard had no payoffs to compare against in that run. It still missed "our plan", but it was not ignoring payoffs it had. When it got messages and actions together, it gave 90 but named J and Z, which is the wrong pair.
Your computed table sounds more like our neat report made by code. That looked great on the first 24 games with Llama at 0.896, but on the new games it fell to 0.636, and Granite got 0.582. So I would test your table on new games also, because ours looked good until then. It may still work better with bigger models or longer transcripts, because we did not test those.
I've seen this with agent workflows too. It's easy to get distracted by what agents say and miss what they actually do. I always prefer logging actions first, then letting the model explain anything unusual.
Yes, "log actions first, then let the model explain the unusual ones" is exactly the order I would use now too. The actions are the hard facts and the model's story sits on top of them not the other way round.
In our test when the model read only the words it was close to guessing. When we counted the actions, a two-line check did much better. So actions first is not just safer, in our case it was the only thing that actually worked. Thanks for sharing how you do it.
This mirrors what happens when people drop an LLM-as-a-judge into multi-agent workflows. The model fixates on the conversational tone in the transcript and completely misses the joint state distribution. If you want to detect boundary crossing or coordinated deviation, an arithmetic invariant on the transition table beats prompt engineering every time. It is deterministic, runs in microseconds, and cannot be smoothed over by a polite cover story.
Yes I actually saw the same thing while running the test. In one game the two players never went to the bar together but the AI still gave them 80 because they kept mentioning each other. So the friendly talk really did fool it and a simple count can't be fooled like that.
But I would not say "every time". I only tested small AI models on one simple game so I can't say this for all systems and the simple check also made the mistakes. It wrongly blamed 3 out of 8 innocent games where actually two players had the same reason to go but no secret plan so the check always gives the same answer but the same answer can still be wrong.
For now I see it like this: use the simple check to find what looks wrong and then let a human or a stronger AI find out why it happened. In your work, which check has been the most useful?
This is a useful reminder that structural invariants beat LLM intuition for detection: instead of asking a model 'does this look collusive,' you compute the joint distribution and check independence — a two-line test that is deterministic and cheap. The same pattern shows up in desktop telemetry: we trust a small statistical check on IPC call patterns far more than an LLM reading a log, because the invariant either holds or it doesn't. The risk is overfitting the detector to one game; generalizing the independence test to real agent tool-use traces is the interesting next step.