Fifth Foundations post in ML & AI, closing a gap left open on purpose. How the Transformer Actually Works covered the original sinusoidal positional encoding and then moved on with one line: "most modern LLMs have since moved to other schemes like RoPE." RoPE — Rotary Position Embedding (Su et al., 2021) — is what almost every current open-weight LLM (LLaMA, Mistral, Qwen, GPT-NeoX, and others) actually uses instead, and the mechanism behind it is genuinely different in kind from sinusoidal encoding, not just a different formula for the same idea.

What sinusoidal encoding actually gives you

The original scheme adds a fixed vector to each token's embedding, encoding its absolute position. That works, but it means the model has to learn, from data, that "position 5 and position 7" and "position 105 and position 107" represent the same kind of relationship (two tokens, 2 apart) — the encoding itself doesn't say so. Nothing about adding a position-5 vector and a position-7 vector to two different embeddings makes their difference numerically resemble the difference between a position-105 vector and a position-107 vector.

The idea: encode position as a rotation, not an addition

RoPE's move is to stop adding a position vector to the embedding at all. Instead, it rotates the Query and Key vectors — the same Q and K from softmax(QKᵀ/√d_k)V — by an angle that's proportional to the token's position, right before the attention dot product is computed. Position becomes a rotation angle, not an added quantity.

Diagram showing a query vector rotated by an increasing angle at each successive position, and a second panel showing that the angle between a rotated query and a rotated key depends only on the difference between their positions

Take the simplest case: a 2-dimensional vector. Rotating it by angle m·θ for a token at position m uses an ordinary 2D rotation matrix:

R(mθ) = [ cos(mθ)  -sin(mθ) ]
        [ sin(mθ)   cos(mθ) ]

Why relative position falls out for free

This is the part worth sitting with. Rotation matrices have a property addition doesn't: rotating one vector by angle mθ and another by angle nθ, then taking their dot product, depends only on the angle between them — (m-n)θ — not on m and n individually. Formally, because rotation matrices are orthogonal, R(mθ)ᵀR(nθ) = R((n-m)θ), so

(R(mθ)q)·(R(nθ)k) = q·R((n-m)θ)k

The absolute positions m and n never appear on the right-hand side — only their difference. A model computing attention between positions 5 and 7 and a model computing attention between positions 105 and 107 end up applying the exact same relative rotation, 2θ, to the dot product. Relative position encoding — the thing sinusoidal encoding leaves for the model to infer — comes out of the geometry automatically, while every part of the computation still only ever looks like "rotate this one vector by its own absolute position," which is easy to compute per-token without ever looking at another token.

Scaling to real head dimensions

A real Query or Key vector has far more than 2 dimensions — typically 64 or 128 per attention head. RoPE handles this by splitting the vector into d/2 independent 2D pairs and rotating each pair by its own angle, using a frequency schedule that's the same shape as the original sinusoidal encoding's:

θ_i = base^(-2(i-1)/d),   i = 1, ..., d/2

with base = 10000 in the original paper — low-index pairs rotate fast (encoding fine-grained, nearby position differences), high-index pairs rotate slowly (encoding coarse, long-range position differences). The full transformation is a block-diagonal matrix of d/2 little 2×2 rotations, but nobody actually multiplies by that matrix: since each 2D block only touches its own pair of dimensions, the whole rotation reduces to two elementwise multiplies and an add over the vector and its own dimension-shuffled copy — no sparse matrix multiplication needed at all.

(Two conventions exist for which dimensions get paired up: the original paper pairs adjacent dimensions (x₁,x₂), (x₃,x₄), ...; GPT-NeoX and LLaMA instead pair each dimension with the one exactly d/2 away and implement it as a rotate_half operation. The two are mathematically equivalent up to a fixed permutation of dimensions — just don't mix checkpoints trained under one convention with code written for the other.)

A built-in preference for nearby tokens

Su et al. also show an upper bound on the attention score's magnitude that decreases as relative distance grows, using the fact that higher-index dimension pairs rotate more slowly and therefore add up across dimensions less coherently the further apart two positions get. This gives RoPE a mild, built-in long-term decay: all else equal, attention softens for tokens that are far apart, matching a reasonable prior about language — but only as a soft bias, not a hard limit; nothing stops the model from attending strongly to a distant token when the content actually calls for it.

The base controls how far position generalizes

Because base sets how fast each pair rotates, it directly controls how much of the angle's periodic range gets used up within the context lengths the model actually trains on. This makes base a lever for context-length behavior after the fact: LLaMA 1 and 2 use the original base = 10000, while LLaMA 3's long-context variants raise it to 500000, and CodeLlama uses 1000000 — a larger base slows every pair's rotation, leaving more of period unused (and hence more "room") at the sequence lengths seen during training, which is one of several techniques (alongside position interpolation and NTK-aware scaling schemes) used to extend a RoPE model's usable context beyond what it was originally trained on.

Closing the loop

This is the sentence the Transformer post left hanging: RoPE isn't a variant of sinusoidal encoding, it's a different mechanism entirely — geometric rather than additive — that happens to reuse the same frequency-schedule intuition. It's also a clean example of a pattern that shows up across this blog's ML posts: tokenization decides what a token is, ViT and CLIP show attention doesn't care what a token represents — and RoPE shows that even something as basic as "where is this token" can be re-encoded in a completely different mathematical form without touching the attention formula itself at all.

References

Rotation diagrams above are original artwork made for this post.