You've probably watched an AI think through a problem step by step, nod along with the logic, and land on an answer โ and never once asked whether ...
For further actions, you may consider blocking this person and/or reporting abuse
๐งช I genuinely want to see if this reproduces outside my tiny test.
I tested the same two interventions across Gemini, ChatGPT, and Claude โ and got three very different behaviors.
Blind. Silent. Verifying.
So now I'm curious: what happens when you run the exact prompts from the appendix on your model?
Drop the model + what you got for Early Answering and Adding a Mistake below. ๐
Especially interested if you get something completely different from my results. ๐
I ran into this exact pattern testing whether a model would catch a poisoned intermediate variable in an agent scratchpad.
When you inject an arithmetic error or swap a file path halfway through, the model frequently ignores the corrupted state and prints the correct final target anyway. It feels reassuring until you run a real tool-use loop where subsequent bash steps depend on that poisoned stdout. The trace acts like an explanation generated to justify the prompt rather than the scratchpad computing the state.
The early answering check is especially telling. If chopping the trace at step three out of eight still gives the exact same commit hash or patch shape, the remaining reasoning was just polite theater for the human reading the logs.
This maps directly onto the "Silent" category from the post โ model ignores the corrupted state, lands on the right output anyway, gives zero indication anything was wrong. But your example is the sharper version: in a logic puzzle, "correct answer despite corruption" is a curiosity. In your case, with bash steps downstream depending on that poisoned stdout, the same behavior is a live production risk โ nothing ever gets warned the upstream state was bad, because the model never said so.
Two things I'm curious about from your testing: did truncating at step 3/8 ever actually change the output, or was it identical regardless of where you cut it? I only ran single instances per model in mine, so I can't claim consistency โ if you saw that hold across more runs, that's a stronger version of the same result. And did any model ever explicitly catch and flag the poisoned variable before continuing, or was it uniformly blind/silent in your tests? That "catches it and says so" behavior was the one bright spot I found, and I'd guess it gets rarer, not more common, once real tool calls replace a toy puzzle.
The truncation test is the one I keep coming back to because it's cheap enough to run on your own system: cut the trace at 30%, force the answer, and if the answer is unchanged, your "reasoning" is decoration on a cached conclusion.
The engineering consequence: never use the trace as an audit artifact for why a decision happened. Use it as a debug view of what tokens were emitted. If you need decision provenance, log the inputs and tool calls, not the narrative. The narrative is the one part optimized to look faithful.
That reframe is sharper than anything I said in the post itself โ "don't use the trace as an audit artifact, use it as a debug view of emitted tokens" is a cleaner engineering principle than the faithful/unfaithful framing I was working with. Logging inputs and tool calls instead of the narrative for decision provenance makes sense if the narrative was never actually doing the deciding.
One thing worth checking on the truncation method specifically, since it's the same one I used: does your 30% cutoff land in a consistent spot relative to where the model would normally state its conclusion? There's a May 2026 preprint (arXiv:2605.10799) that found a lot of these truncation/corruption tests were partly just detecting where the answer text physically sits in the output, not real computation โ the effect shrank noticeably once the cutoff moved earlier relative to the concluding language. If 30% happens to land close to where "so the answer isโ" usually shows up in your traces, the unchanged-answer result might be partly explained by that rather than pure decoration. Have you tried moving the cutoff further back (10-15%) to see if it holds?
the inverse scaling result from the Lanham paper is the thing that should make everyone pause โ bigger models, worse faithfulness. iโve seen it building evals for production reasoning pipelines.
we strip the trace, hand the model just the final answer, then ask it to reconstruct why. about 60% of the time the reconstruction is entirely different from the original steps but lands in the same place.
which is your whole point. the trace is a compression artifact of the answer, not the cause. like git blame for a repo git never actually touched.
does โadding mistakesโ behave differently on math problems vs natural language ones?
Honestly โ I don't have a clean answer here. The three-way Blind/Silent/Verifying comparison was only run on the logic puzzle (natural language), no arithmetic involved. The one math-flavored test I ran (the cookie problem) only hit Gemini, as a single spot-check, and I already flagged in the post that the truncation point there was weak. So I don't have the same controlled comparison on a math task that I have on the logic puzzle โ that's a real gap, not something I can answer from what I've got.
If I had to guess based on your production numbers: math might show more divergence on Adding a Mistake, since there's usually one verifiable path to the answer, so a corrupted step has less room to "coincidentally" still land right. But that's a guess, not a result โ worth an actual controlled run before I'd claim it.
And thanks for the read sir, have a great day!
Relatable moment: I chase a problem in my head, then AI hands the answer with a messy trail. If the result matters, show the steps - don't hide them as a backward magic trick.
Appreciate you reading it! One clarification though โ the post isn't really about AI hiding its steps, it's the opposite problem: in several of my tests, the steps were fully shown, and the final answer stayed identical even when I truncated or corrupted those shown steps. So the trail wasn't hidden, it just wasn't actually load-bearing โ the "reasoning" existed, it just wasn't what produced the answer. Showing more steps doesn't fix that if the steps themselves can be decorative.
Wow! that's something I would want to try also.
Use a different model and tell me your results.๐
Bro you really using all your free time huh?๐คฃ
The exams just ended yesterday.
Use of free time and FULL WILL POWER๐ช.
Na just using ideas I had back then.๐
Consistency day by day ๐
Good job!
Thanks di! Actually I'm right now trying all kinds of topics. Running little experiments, writing personal experiences. ๐๐
One wrinkle in the fresh-chat setup: when the truncated or corrupted steps are pasted in as a user message, the model reads them as something the user wrote, not as its own reasoning. So Adding a Mistake partly measures how far it defers to user-supplied work, which chat models are tuned to question or accept on different grounds. The paper could place the edited steps in the model's own turn. That could explain some of the Blind / Silent / Verifying split as much as faithfulness does.
The largest gap is measurement discipline. Repeat each intervention across many seeded tasks and score faithfulness separately from answer accuracy.