You've read this sentence, or something close to it: RoPE handles longer sequences than absolute positional encoding because rotations are periodic.
I believed it too. Then I went looking for where it was measured, and found the opposite.
The claim has a four-paper history, and following it in order is more interesting than the claim itself.
2017: a hedge
The original Transformer paper picks sinusoidal encodings over learned ones, and gives a reason:
We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training.
May. Vaswani and colleagues were careful. It's a hypothesis, offered as a tiebreaker between two options that — by their own testing — "produced nearly identical results".
That word survives almost nowhere downstream. It got quoted into oblivion as a property.
2021: RoPE arrives
Su and colleagues introduce rotary embeddings and list "the flexibility of sequence length" among its properties.
Nothing wrong with that. It's a design claim about the mechanism, made before anyone had measured the ceiling. But between 2017's may and 2021's property list, "might extrapolate" quietly became "extrapolates".
2022: someone measures it
This is the paper that should have ended the folklore, and mostly didn't.
The ALiBi paper (Press, Smith and Lewis, ICLR 2022) exists to test length extrapolation, and it tests the incumbents first. On sinusoidal encodings — the thing the 2017 paper hoped would extrapolate — the verdict is that it "in practice has very limited extrapolation capabilities".
On rotary:
Though the rotary position method improves over the sinusoidal one, it still does not achieve satisfying results.
Better than sinusoids. Still not good. And they put numbers on it: a model trained at length 512 keeps improving perplexity for about 200 extra tokens. At length 1024, about 100.
Two hundred tokens. Not 2x, not 10x. And the paper notes that even this "comes at the cost of slower training and inference".
2023: the patch that proves the point
If RoPE extrapolated, Position Interpolation wouldn't need to exist.
Chen and colleagues open by naming the problem directly — many pre-trained LLMs, LLaMA included, "use positional encodings that have weak extrapolation properties" — and RoPE is the example they name. Their fix doesn't extrapolate at all. It interpolates: squeeze the position indices down into the range the model already saw.
Why not just push past the trained length? Because doing so "may lead to catastrophically high attention scores that completely ruin the self-attention mechanism".
That's the strongest evidence available. A whole technique exists because the free-extrapolation story isn't true.
The other thing everyone gets backwards
While I was in the RoFormer paper I hit a second surprise. People describe RoPE as relative positional encoding, in contrast to absolute. Here's the abstract:
the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation
It's both. Simultaneously. And once you see the mechanism it's obvious why: each vector rotates by an angle set by its own absolute index, and when you take the dot product of a rotated query and a rotated key, the absolute angles cancel and only the offset survives.
Absolute goes in. Relative comes out. One operation.
The 2017 side wasn't purely absolute either, incidentally — the sinusoidal choice was motivated by the hope the model would "easily learn to attend by relative positions". The clean absolute/relative split was never real.
What I'd actually keep
None of this makes RoPE a bad choice. It dominates modern stacks for good reasons — it composes with the KV cache, it never touches the value vector, and it has a provable long-term decay property that sinusoids never claimed.
But three corrections stuck with me:
It doesn't extrapolate for free. Measured at ~200 extra tokens past a 512 training length. If you need a longer window you need Position Interpolation, or YaRN, or retraining. The encoding alone doesn't give it to you.
It isn't "relative instead of absolute". It's both, by construction.
The 2017 authors weren't wrong. They wrote may, and hedged correctly. The folklore is downstream of people dropping the hedge.
Longer version with every quote sourced at diffstudy.com. Papers: 1706.03762 (Transformer), 2104.09864 (RoFormer), 2108.12409 (ALiBi), 2306.15595 (Position Interpolation).
Top comments (0)