RoPE, position as a rotation

Day 1 added position by summing a position vector into each token vector before the first block. Every current open decoder does it differently. It leaves the token vectors alone and rotates the queries and keys inside every attention layer. That's RoPE, and it was the first thing Day 2 swapped in.

Attention cares about distance, not absolute position

Whether "it" should read "cat" depends on how far back "cat" is and what's in between, not on whether the pair sits at positions 3 and 7 or 503 and 507. A learned position table gives each absolute position its own vector, so the model has to learn separately that the pairs (3, 7) and (503, 507) are alike. And past its last row it has no vector at all.

What we actually want is a score $q_m \cdot k_n$ that depends on the content of both tokens and on the offset $m - n$, and nothing else about $m$ and $n$.

Rotate each pair of channels by an angle that grows with position

Split the query's $d$ channels into $d/2$ adjacent pairs, and treat each pair as a point on a plane. At position $m$, rotate pair $i$ by the angle $m\,\theta_i$:

Do the same to the key at position $n$. The dot product of two rotated 2D vectors only depends on the angle between them. So rotating $q$ by $m\theta$ and $k$ by $n\theta$ is the same, inside the dot product, as rotating nothing and turning $k$ by $(n - m)\theta$:

So the score depends on the content of $q$ and $k$ and on $n - m$. That's exactly the property we wanted, and it holds in every pair, each one at its own speed.

So what happened ?

• "Shift both by" turns both arrows by the same angle. The angle between them is fixed, so the dot product is fixed to four decimals.
• Changing one position changes the angle between them, and the dot product follows.
• The lengths of $q$ and $k$ never change. A rotation doesn't scale anything, so RoPE carries position without changing how big a score can get.
• Pair 0 turns a full radian per position and wraps around the circle about every 6 positions, so it can tell neighbors apart and nothing more. Pair 3 turns so slowly that it separates positions dozens of tokens apart.

The Day 2 paper exercise traced the same thing by hand for pair 0, where $\theta = 1$:

Tables once per forward pass, rotation per layer

The cosines and sines only depend on the length and the head width, so octlm builds them once per forward pass and every layer reuses them. Nothing is learned, and there's no maximum length.

def rope_tables(
    length: int, head_size: int, scale: float, device: torch.device, base: float = ROPE_BASE
) -> tuple[Tensor, Tensor]:
    index = torch.arange(0, head_size, 2, device=device, dtype=torch.float32)
    inverse_frequency = 1.0 / torch.pow(base, index / head_size)
    position = torch.arange(length, device=device, dtype=torch.float32) / scale
    angle = torch.outer(position, inverse_frequency)
    return angle.cos(), angle.sin()

scale is position interpolation, which comes up below. The rotation itself uses the paper's elementwise form on each adjacent pair:

def apply_rope(x: Tensor, cosine: Tensor, sine: Tensor, split: bool = False) -> Tensor:
    """Rotate each adjacent channel pair of [batch, heads, length, head_size] by its angle."""
    if split:
        cosine = torch.cat((cosine, cosine), -1).to(x.dtype)
        sine = torch.cat((sine, sine), -1).to(x.dtype)
        left, right = x.chunk(2, -1)
        return x * cosine + torch.cat((-right, left), -1) * sine
    pairs = x.float().unflatten(-1, (-1, 2))
    left, right = pairs[..., 0], pairs[..., 1]
    rotated = torch.stack((left * cosine - right * sine, left * sine + right * cosine), dim=-1)
    return rotated.flatten(-2).type_as(x)

RoPE applies to queries and keys only, never values. The values carry content to be mixed, and position has already done its job by shaping the scores. The math runs in float32 even when the model runs in half precision, and the result is cast back.

Two tests hold the implementation to the two properties above, test_rotation_preserves_length and test_scores_depend_only_on_relative_distance. A test on this site checks the page's rope_tables and apply_rope ports against the Python functions.

How far past the training length does it hold ?

day-wise.md framed RoPE as "extrapolation to 2K, 4K, 8K". The Position Interpolation paper says otherwise, past the trained length raw RoPE can fail badly, with perplexity above 1,000. So the note wrote the opposite hypothesis and tested that.

Setup, two models trained at a 512-token context on the Day 2 corpus, one with sinusoidal positions and one with RoPE. The control is sinusoidal because a learned table can't run past 512 at all. Both were then evaluated at five lengths. A third column evaluates the RoPE model with interpolation, which divides every position by (evaluation length ÷ 512) so the angles stay inside the range training saw.

So what happened ?

• RoPE beats sinusoidal at every length, by 0.76 bits per byte at 2,048.
• Both models get better from 512 to 2,048. A longer window gives each prediction more context, and fewer positions sit at the start of a block with barely any context.
• Both turn back up after 4,096, which is 8 times the trained length.
• Nothing collapses. At 16 times the trained length RoPE is worse than at 512 but nowhere near the failure the paper reports. The note's explanation is that the paper's models trained at 2,048, so they met way bigger absolute distances than a model trained at 512 meets at 8,192.
• Interpolation only helps at 8,192, by 0.032. From 1,024 to 4,096 it costs 0.02 to 0.04, because squeezing the angles below what training saw hurts while raw extrapolation still works fine.

Decision, keep RoPE. Only interpolate past 4 times the trained length, and re-check that crossover after any change to the training context. The paper's other half, fine-tuning for 1,000 steps at the longer length, wasn't tested.

What we didn't build, and why

• Fine-tuning at the longer length. The Position Interpolation result depends on it. It belonged to a later phase, and that phase got dropped.
• NTK-aware or YaRN scaling. Both change the base frequency instead of dividing positions. Plain interpolation already answered the question at our lengths.
• A different base. octlm keeps 10,000 for its own models. Qwen's config sets 1,000,000, and Day 6 reads it from config.json, along with Qwen's split-half channel pairing.