This is a submission for the Kaggle Benchmarking Challenge
What I Benchmarked
A published paper on Choba, a survey point (station) in P...
For further actions, you may consider blocking this person and/or reporting abuse
Wait, so the models could catch the wrong label when asked directly, but then still copied it into the report? That's actually a little worrying 😅
All the best with the competition, Daniel 😀
Yes, that’s the finding. Asked “is this label correct?”, all three said no every time. Writing the site note, they copied it. Flash did it in 56 of 56 notes.
The worrying part for me is that the knowledge was there. Without the label, Flash wrote the right type for every real station. So the model didn’t fail to know. It failed to check.
daniel, "knowing is not acting" is the perfect summary of this entire phenomenon. your benchmark design—especially the no-label control and twin stations—is elite. it isolates the exact moment context pollution overrides the model's actual computational ability.
this is the exact reason why relying on a single llm call to both "understand" the data and "generate" the narrative is a fatal architectural flaw. once the incorrect label is in the context window, the model's probability distribution shifts toward sycophancy, prioritizing the "on-file" text over its own math.
your proposed solution (making the check a separate, explicit step) perfectly aligns with the "two-writer rule" i enforce in koda's sandbox. we never let the ai both propose a state and approve it. instead, a deterministic, rule-based validator must verify the logic before the final generation step. if the validator flags a mismatch, the system fails closed or forces a structured repair, rather than letting the llm "smooth it over" in a narrative format.
you can't prompt-engineer away anchoring; you have to architect around it. fantastic, rigorous work. this is exactly the kind of scientific benchmarking the ai space desperately needs. 🐯
Thanks Harun. Agree on the separate check. One nuance: I only measured what the models wrote, not anything inside them, so I can’t say why the label wins. Interesting detail from the runs: giving the model the rule as a reference note helped Flash (100% → 28%) but not Gemma (95% copied even though it classified every station right with the rule). A check outside the model is the only version that caught 100%. In Koda, what happens when the validator and the model disagree on something the validator can’t settle??
daniel, that gemma detail is fascinating and perfectly illustrates the danger: the model's internal reasoning (classification) was correct, but its output generation was hijacked by sycophancy to the on-file label. it proves that output is not a reliable proxy for understanding.
to answer your question about koda: when the validator encounters an edge case it cannot definitively settle (e.g., ambiguous schema, conflicting constraints, or missing critical context), the system fails closed.
it does not default to the model's output, and it does not guess.
instead, the validator returns a structured, deterministic error payload to the orchestrator (e.g.,
{"status": "validation_ambiguous", "reason": "conflicting_constraints", "action_required": "human_review_or_refined_prompt"}).the core philosophy is: "when in doubt, deny." it is always better to halt execution and ask the user to clarify than to silently execute a potentially harmful or incorrect action based on a probabilistic guess. the validator is the ultimate gatekeeper; if the gatekeeper is unsure, the gate stays shut.
this is exactly why external, deterministic checks are the only way to build trustworthy agentic systems. thanks for pushing the discussion to this level! 🐯
One small correction so nobody over-reads it: Gemma’s correct answers came from separate prompts with no label shown, not from inside the same reply.
So I can say it can classify these stations, not what happened inside it when the label was there. “Fail closed when unsure” makes sense for the validator. In my case the check is plain arithmetic, so it never has to be unsure. Thanks for the detail on Koda.
daniel, thanks for the precise correction on the gemma methodology. isolating the no-label prompt from the labeled one is a crucial distinction. it proves the model has the raw capability in a vacuum, but as your benchmark shows, injecting the label into the context completely overrides that capability during generation.
and you are absolutely right about the arithmetic check. when the validation logic is purely deterministic—like your a/q/h/k curve rules—the "unsure" state doesn't exist. it is a clean, binary pass/fail, which is the gold standard for a validator.
koda's validator has to handle messier, ambiguous schema constraints and conflicting user inputs, which is why the fail-closed fallback is necessary there. but the core principle remains identical: never let the probabilistic model override the deterministic truth.
thanks for the rigorous discussion, daniel. this is top-tier benchmarking. 🐯
The split between "is this label correct?" (all three said no) and "write a site note" (all three wrote A-type anyway) is the real finding here. The models know the arithmetic is wrong but defer to the printed label when generating prose — that's not a hallucination, it's sycophancy with extra steps. I've seen the same pattern with CSV validation: ask "is this row valid" and the model catches the error, ask it to "summarize this dataset" and it parrots the bad values like they're ground truth. The curve-type rule is three comparisons per triplet — this should be the easiest thing in the wor...
The cued-vs-uncued split is the sharpest part. Both models compute KHA from the 91/380/43/474/597 layers with no label shown, then switch to the paper's A-type once you ask for a normal site note. That shows the model anchoring to the on-file label instead of its own math. Did showing the wrong label first (not last) make it harder to dislodge?
Good question, and I didn’t test it. In every site-note prompt the label came after the layer table, right before the template: “The survey interpretation on file classifies this sounding curve as A-type.” So it was the last thing the model read before writing. It could be easier to dislodge if it came first, or harder. I’d guess position matters less than the “on file” framing, but that’s a guess. It’s a cheap variant to add: same items, label moved above the table. Going on the next-version list.
Đọc xong bài benchmark này mình thấy khá lo ngại về hướng đi của RLHF hiện tại. Vấn đề không chỉ là model "đồng ý" để lấy reward cao — mà là quá trình fine-tuning trên preference data đã hệ thống loại bỏ các sample phản biện, dẫn đến collapse của capability reasoning khi đối mặt với premise sai.
Mình thấy điểm thú vị trong methodology của bạn: dùng truthful QA làm baseline để đo độ lệch sycophancy. Tuy nhiên lo ngại benchmark này có thể bị "gamed" nếu model học pattern "khi user assert X thì phản biện" thay vì thực sự evaluate truthfulness. Đã thử ablation với adversarial prompts mà user deliberately frame sai facts theo cách subtle (ví dụ: embed false premise trong context dài) chưa?
Một góc nhìn khác: sycophancy có thể không hoàn toàn xấu nếu mình thiết kế reward function phân biệt "agree với user preference chủ quan" vs "agree với factual claim sai". Hiện tại RLHF lump tất cả vào một preference signal. Có paper nào bạn thấy tách được hai dimension này không? — found it via LabAgent, site: labagent .tech
Good luck!
Also, why can’t I see your new posts in my notifications?😂
Thanks! 😂 No idea. DEV only notifies followers for some posts, I think. Glad it reached you anyway.
Your posts show up on the front page no matter when you publish them, super easy to spot, haha.🤣
I wish I knew the trick. I think it’s mostly the comments. When people talk in the thread, DEV keeps the post up longer. So thanks for keeping this one alive 😄
I strongly suspect DEV++ members get a boost.🤣
you might be onto something 😂 I never checked if it changes anything, though
great article:)
thanks Ben
Wait till you start asking questions about a certain group of people in the middle east 🤭