Positional Encoding: RoPE & ALiBi

"The model loaded the checkpoint" and "The checkpoint loaded the model" share the exact same words, but reverse who loaded what. Content-only, unmasked self-attention follows each word wherever it moves; it has no slot index that marks the subject as preceding the object.

Vision transformers expose that gap with image patches, and text models face the exact same dilemma. The self-attention mechanism compares content, not sequence position. Predict what happens when the token rows for "model", "loaded", and "ckpt" are shuffled.

The answer gives us our first foundational diagnostic: unmasked attention shuffles its outputs right alongside its inputs. That's permutation equivariance. A position signal breaks that symmetry inside a transformer. It can enter as an additive vector merged into embeddings, a RoPE rotation on queries and keys, or ALiBi's linear distance bias on attention logits.

Then we'll stress that same principle at long context lengths. A mathematical formula can produce a value at position 32,000 even when pretraining stopped at 4,096. That calculation is valid arithmetic, but it isn't evidence that the model can actually retrieve a buried passkey there.

One boundary stays fixed throughout: a position method can rank permitted keys, while a causal mask must still hide future keys. A reliable diagnostic test reorders the sequence, verifies the permitted scores, and confirms that forbidden positions remain forbidden.

First, watch attention lose the order

Before adding any position feature, make one prediction. If every token vector stays the same but the rows move, should content-only attention preserve the original row order, or should its outputs move with them?

Standard self-attention computes:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

For an input sequence of $N$ tokens represented as matrix $X \in \mathbb{R}^{N \times d_{\text{model}}}$, linear projection matrices $W_Q, W_K \in \mathbb{R}^{d_{\text{model}} \times d_k}$ and $W_V \in \mathbb{R}^{d_{\text{model}} \times d_v}$ produce $Q = X W_Q$, $K = X W_K$, and $V = X W_V$. The dot product $Q K^T$ forms an $N \times N$ score matrix. Dividing by $\sqrt{d_k}$ controls variance under the usual independent, unit-variance component assumptions. Row-wise softmax produces attention weight matrix $A$, which mixes values into output $Y = A V \in \mathbb{R}^{N \times d_v}$. No term in that formula reads "first", "second", or "third."^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}

Now apply an arbitrary $N \times N$ permutation matrix $P$. Permuting the input rows yields $X' = P X$. Because matrix multiplication is associative, the projections reorder identically:

$Q' = (P X) W_Q = P Q, \quad K' = (P X) W_K = P K, \quad V' = (P X) W_V = P V$

Compute the attention logits under permutation:

$S' = \frac{Q' (K')^T}{\sqrt{d_k}} = \frac{(P Q)(P K)^T}{\sqrt{d_k}} = P \left(\frac{Q K^T}{\sqrt{d_k}}\right) P^T = P S P^T$

Row-wise softmax distributes cleanly over row and column permutations:

$A' = \text{softmax}(P S P^T) = P \text{softmax}(S) P^T = P A P^T$

Finally, multiply by permuted values $V' = P V$:

$Y' = A' V' = (P A P^T)(P V) = P A (P^T P) V$

Because any permutation matrix is orthogonal ($P^T P = I$), the inner permutation matrices cancel out completely:

$Y' = P A I V = P (A V) = P Y$

That equality, $\text{Attention}(P X) = P \text{Attention}(X)$, defines permutation equivariance for this deterministic, unmasked calculation. Shuffling input rows shuffles output rows; the output matrix is not unchanged. Token-wise projections, residual additions, and token-wise normalization preserve this property. Dropout needs to be disabled or its sampled mask permuted consistently for an exact numerical comparison.

Use a three-token slice of our running example, with one 2D vector per token:

Permutation $P = [2, 0, 1]$ rewrites the sequence as ckpt, model, loaded. Unmasked attention on the shuffled rows produces exactly the shuffled original output: $Y' = P Y$. The block preserves content relationships, but remains blind to command order.

The small check below recomputes attention after the permutation. Read True as a diagnosis: no hidden slot number entered the calculation.

import math

tokens = {
    "model": [1.0, 0.0],
    "loaded": [0.0, 1.0],
    "ckpt": [1.0, 1.0],
}
order = ["model", "loaded", "ckpt"]
permutation = [2, 0, 1]

def attention(rows: list[list[float]]) -> list[list[float]]:
    scores = [
        [sum(q_i * k_i for q_i, k_i in zip(q, k)) / math.sqrt(2) for k in rows]
        for q in rows
    ]
    outputs = []
    for row in scores:
        maximum = max(row)
        exponentials = [math.exp(value - maximum) for value in row]
        normalizer = sum(exponentials)
        weights = [value / normalizer for value in exponentials]
        outputs.append([
            sum(weight * value[column] for weight, value in zip(weights, rows))
            for column in range(2)
        ])
    return outputs

baseline = attention([tokens[name] for name in order])
shuffled = attention([tokens[order[index]] for index in permutation])
expected = [baseline[index] for index in permutation]

same_output_up_to_permutation = all(
    abs(actual - target) < 1e-12
    for actual_row, target_row in zip(shuffled, expected)
    for actual, target in zip(actual_row, target_row)
)
print(same_output_up_to_permutation)
print([round(row[0], 3) for row in baseline])
print([round(row[0], 3) for row in shuffled])

True
[0.802, 0.599, 0.752]
[0.752, 0.802, 0.599]

A causal mask changes the symmetry by removing future keys from view. In a decoder, token $i$ sees keys $j \le i$, including itself. If token rows move while that triangular mask stays fixed, the score calculation generally changes. Permuting the mask along with the rows would preserve equivariance, but would represent a different visibility pattern.

Haviv et al. trained NoPos causal language models and found competitive perplexity in their tested 125M-to-1.3B settings. Probes recovered approximate absolute positions; counting visible predecessors is their proposed explanation, not a proven counter built into softmax.^{[2]Reference 2Transformer Language Models without Positional Encodings Still Learn Positional Information.https://arxiv.org/abs/2203.16634} Identical value rows illustrate the limit: their weighted average stays identical even when one query sees more rows. The mask permits position learning; it does not guarantee it for every input or model.

Parallel prefill still uses the decoder's causal mask. Computing many query rows at once does not make them bidirectional. A BERT-style masked-language-model objective is different: masking some input words does not itself impose triangular attention visibility. Without any position or order-dependent signal, an unmasked block stays permutation equivariant and can still learn content relationships. Haviv et al.'s bidirectional NoPos language-model experiment degraded sharply; that is not a claim that every permutation-equivariant task fails.

Absolute encodings, RoPE, and ALiBi insert position at different points in the block. We'll compare them separately. Architectures can combine position mechanisms; reproducing a checkpoint means following its actual design.

The table is a map of those insertion points. For each row, ask what downstream quantity can first "feel" position.

When debugging a position mismatch, inspect the first tensor that changed: the embedding, projected Q/K, or attention score matrix. A method added accidentally changes the checkpoint's computation even if another position mechanism is already present.

The first map entry adds position before any Q, K, or V projection. Start there, then we'll move the signal deeper into attention.

Absolute position: give every slot a multi-speed fingerprint

Nearby slots need distinguishable labels, while far slots need a slower signal that tracks broad location. The original Transformer meets both needs with several paired waves added to each token embedding.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}

At position 0, every sine starts at 0 and every cosine at 1. Move to position 2 and the fastest pair has moved two radians, while slower pairs barely move. The combined phases form a position fingerprint before the network sees Q, K, or V.

For position $\text{pos}$ and dimension $i$, with $d_{\text{model}}$ the embedding width:

$PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

$PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

The paired waves carry one useful algebraic property. For a pair ordered as $[\sin(p\omega),\cos(p\omega)]$, shifting by $k$ gives

The matrix depends on the offset, not the starting position. Its sign convention follows the sine-first ordering; RoPE below uses ordinary Cartesian $[x,y]$ pairs. This algebra gives later layers a route to learning relative relationships. Adding the vectors doesn't itself force attention to depend only on relative offset.

The generator below makes the dial picture concrete with an 8-by-8 table. Before running it, predict row 0 and the first pair at row 2.

import math

def sinusoidal_positional_encoding(max_len: int, d_model: int) -> list[list[float]]:
    if (type(max_len) is not int or type(d_model) is not int
            or max_len < 0 or d_model <= 0 or d_model % 2):
        raise ValueError("max_len must be nonnegative and d_model positive and even")
    table: list[list[float]] = []
    for pos in range(max_len):
        row = [0.0] * d_model
        for i in range(0, d_model, 2):
            frequency = math.exp(-(math.log(10000.0) * i) / d_model)
            angle = pos * frequency
            row[i] = math.sin(angle)
            row[i + 1] = math.cos(angle)
        table.append(row)
    return table

pe = sinusoidal_positional_encoding(max_len=8, d_model=8)
print(len(pe), len(pe[0]))
print([round(x, 3) for x in pe[0]])
print(round(pe[2][0], 3), round(pe[2][1], 3))

8 8
[0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0]
0.909 -0.416

The dials have different wavelengths. With $d_{\text{model}}=512$, a pair whose first dimension is $r$ has wavelength $\lambda_r = 2\pi\cdot10000^{r/d_{\text{model}}}$. Here $r=2i$: the pair index and the dimension index aren't interchangeable.

import math

d_model = 512
for first_dimension in [0, 64, 510]:
    angular_frequency = 10000 ** (-first_dimension / d_model)
    wavelength = 2 * math.pi / angular_frequency
    print(first_dimension, round(wavelength))

0 6
64 20
510 60611

The waves are easier to read as lanes than as a matrix:

What addition guarantees, and what it leaves learned

Addition gives the attention dot product several ways to use position. Let token $m$ have content vector $x_m$ and position vector $p_m$, and let token $n$ have content $x_n$ and position $p_n$. The network projects the combined sums:

$q_m = (x_m + p_m) W_Q, \quad k_n = (x_n + p_n) W_K$

Expanding their unscaled attention dot product yields four terms:

$q_m k_n^T = (x_m + p_m) W_Q W_K^T (x_n + p_n)^T = \underbrace{x_m W_Q W_K^T x_n^T}_{\text{content-content}} + \underbrace{x_m W_Q W_K^T p_n^T}_{\text{content-position}} + \underbrace{p_m W_Q W_K^T x_n^T}_{\text{position-content}} + \underbrace{p_m W_Q W_K^T p_n^T}_{\text{position-position}}$

Here vectors use the row convention of $Q=XW_Q$; $q_m k_n^T$ is a scalar. The familiar column-vector notation is $q_m^T k_n$ after transposing both vectors consistently.

The expansion separates three questions:

1. Is the position-only term relative? If $W_Q W_K^T=I$, each sine/cosine pair contributes $\cos((m-n)\omega)$, so that term depends only on offset. For a general learned product, shared-shift invariance is not guaranteed. More broadly it requires $M_k^T A M_k=A$ for column vectors with bilinear matrix $A$; an exact block-diagonal rotation is not the only solution. Even setting $W_Q=W_K$ does not ensure this condition.
2. Do the cross-terms vanish? No. Content can interact with absolute position, which may help some tasks or hurt shift generalization. Their existence does not prove a particular learned spurious affinity.
3. Can we supply an unseen position? A learned table with $L_{\max}$ rows has indices $0$ through $L_{\max}-1$. Position $L_{\max}$ is already outside it; extending or remapping rows changes the design and needs adaptation.^{[3]Reference 3BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805[4]Reference 4GPT-2 Source Implementation.https://github.com/openai/gpt-2/blob/master/src/model.py} A sinusoidal formula has no finite table boundary, but a cached implementation may still need a larger buffer. Its bounded sine/cosine coordinates cannot by themselves guarantee quality or force exploding logits. Press et al. measured poor length extrapolation for their sinusoidal language models.^{[7]Reference 7Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.https://arxiv.org/abs/2108.12409}

Vaswani et al. tested both and reported "nearly identical results" (Table 3, row E). Sinusoids were attractive because their formula can be evaluated at unseen indices, but definition isn't generalization.

Why add instead of concatenate? Put two $d$-wide vectors side by side and the next layer sees width $2d$. Unless you project back down, square matrices such as $W_Q$, $W_K$, and $W_V$ then grow about 4x. Addition keeps width $d$, but it also gives later projections one mixed vector rather than separate content and position channels.

Absolute methods leave us with three questions. Does quality hold beyond the trained length? Can the model infer an offset from two absolute vectors? Is mixing content and position in one stream a good trade? Relative methods move the distance signal into attention itself, which is the next experiment.

RoPE: make a shared offset visible inside QK

Now move the position signal past the embedding stage. Keep one query vector and one key vector fixed, then place them at positions $(m, n)$. If both positions shift forward by 100 slots, should their attention routing score change?

Rotary Position Embedding (RoPE) rotates query and key vectors in 2D subspaces.^{[5]Reference 5RoFormer: Enhanced Transformer with Rotary Position Embedding.https://arxiv.org/abs/2104.09864} For fixed unrotated Q/K and one shared frequency schedule, its explicit position dependence reduces to the relative offset $n-m$. Earlier layers can already make those unrotated vectors depend on context and position. The identity below is a statement about this rotation step, not translation invariance of an entire trained model.

The 2D rotation matrix and relative offset invariance

To encode position into a 2D coordinate pair $(q_{2i}, q_{2i+1})$ at sequence index $m$, RoPE applies a 2D rotation matrix $R_{\theta_i, m}$:

For a full head dimension $d_k$ (where $d_k$ is even), RoPE constructs a block-diagonal rotation matrix $R_{\Theta, m} \in \mathbb{R}^{d_k \times d_k}$ consisting of $d_k / 2$ independent 2D rotation blocks:

The angular frequencies follow a geometric progression parameterized by rotary dimension $d_{\text{rotary}} \le d_k$ and base $b$:

$\theta_i = b^{-2i/d_{\text{rotary}}}, \quad i \in \left\{0, 1, \dots, \frac{d_{\text{rotary}}}{2} - 1\right\}$

For partial RoPE, build these blocks only for the even-sized rotary prefix and use an identity transform on the remaining coordinates. The rotation preserves the full head's norm. The original schedule uses base $10000$; checkpoint bases and frequency rescaling vary. Raising the base slows pairs with $i>0$, while the first frequency $\theta_0=1$ stays unchanged. Llama 3.1 uses a base of $500{,}000$ plus a piecewise frequency transform, discussed below.^{[8]Reference 8Llama 3.1 reference rotary embedding and frequency scalinghttps://github.com/meta-llama/llama-models/blob/f45cdfd624b98b6655540f7101d8d9cb432e631c/models/llama3_1/reference_impl/model.py}

Now examine the inner product between query $q^{(m)} = R_{\Theta, m} q$ and key $k^{(n)} = R_{\Theta, n} k$:

$(q^{(m)})^T k^{(n)} = (R_{\Theta, m} q)^T (R_{\Theta, n} k) = q^T (R_{\Theta, m}^T R_{\Theta, n}) k$

Because $R_{\Theta, m}$ is block-diagonal, evaluate the product block-by-block. For one 2D pair at frequency $\theta$:

Multiplying out the matrix entries using standard trigonometric difference identities yields:

Combining those terms proves the fundamental RoPE identity:

$R_{\theta, m}^T R_{\theta, n} = R_{\theta, n-m}$

With fixed Q/K and the same frequencies for both sides, absolute indices cancel from the explicit rotation:

$(R_{\Theta, m} q)^T (R_{\Theta, n} k) = q^T R_{\Theta, n-m} k$

Shift both indices by 1,000 while keeping Q/K and the frequency schedule fixed. The rotation matrix $R_{\Theta, (n+1000) - (m+1000)} = R_{\Theta, n-m}$ gives the same score in exact arithmetic. Finite-precision angles can introduce numerical differences.

Complex number formulation and implementation

Identifying each 2D coordinate pair $(q_{2i}, q_{2i+1})$ with a complex number $z = q_{2i} + i q_{2i+1} \in \mathbb{C}$ reveals the rotation as complex multiplication:

$q^{(m)} = (q_{2i} + i q_{2i+1}) e^{i m \theta_i}, \quad k^{(n)} = (k_{2i} + i k_{2i+1}) e^{i n \theta_i}$

The Euclidean inner product of two real 2D vectors equals the real part of their complex inner product $\text{Re}(z_1 z_2^*)$:

$\langle q^{(m)}, k^{(n)} \rangle = \text{Re}\left( (q e^{i m \theta_i}) (k e^{i n \theta_i})^* \right) = \text{Re}\left( q \bar{k} e^{i (m - n)\theta_i} \right)$

In real tensor arithmetic, avoid building the sparse $d_k \times d_k$ matrix. Instead, apply the rotation via elementwise products:

$R_{\theta_i, m} \begin{pmatrix} q_{2i} \\ q_{2i+1} \end{pmatrix} = \begin{pmatrix} q_{2i} \\ q_{2i+1} \end{pmatrix} \odot \begin{pmatrix} \cos(m\theta_i) \\ \cos(m\theta_i) \end{pmatrix} + \begin{pmatrix} -q_{2i+1} \\ q_{2i} \end{pmatrix} \odot \begin{pmatrix} \sin(m\theta_i) \\ \sin(m\theta_i) \end{pmatrix}$

This operation has two core properties:

1. Norm preservation: An orthogonal rotation satisfies $\|R_{\Theta,m}q\|_2=\|q\|_2$ in exact arithmetic. Individual coordinate variances can change; preservation of vector length does not prove unchanged per-coordinate statistics. An extra YaRN amplitude factor is a separate scaling operation.
2. Routing versus aggregation: Standard RoPE here rotates Q/K and leaves V unrotated. That introduces position into routing without directly rotating the retrieved values. Other trained architectures can modify values; adding an unexpected V rotation to this checkpoint would change its computation.

import math

def rotate_2d(vec: list[float], pos: int, theta: float) -> list[float]:
    cosine, sine = math.cos(pos * theta), math.sin(pos * theta)
    x, y = vec
    return [x * cosine - y * sine, x * sine + y * cosine]

def dot(left: list[float], right: list[float]) -> float:
    return sum(x * y for x, y in zip(left, right))

query = [0.8, 0.6]
key = [0.7, 0.5]
theta = 0.5
score_a = dot(rotate_2d(query, 5, theta), rotate_2d(key, 2, theta))
score_b = dot(rotate_2d(query, 105, theta), rotate_2d(key, 102, theta))
print(round(score_a, 6), round(score_b, 6))

0.040884 0.040884

The output confirms the shared-shift prediction. It also marks the limit of this toy proof: a trained model’s context window depends on content-dependent Q and K vectors across many layers, not on this identity alone.

Use a unit-vector walk-through to isolate the phase term. Let $q = k = [1, 0]$ and $\theta = 0.5$.

• Query at $m=2$, key at $n=1$: offset $d = n-m = -1$, score $\cos(-0.5) \approx 0.878$.
• Query at $m=102$, key at $n=101$: same offset, same score.

At offset $-2$, the same pair scores $\cos(-1.0) \approx 0.540$. Don’t turn that comparison into “farther is always weaker.” Cosine phases can rise again, and a real head sums many frequencies.

Take a head with $d_k=d_{\text{rotary}}=8$, containing four pairs. With the standard base-$10000$ schedule, their frequencies are $1$, $0.1$, $0.01$, and $0.001$. For $q=k=[1,0]$ in every pair and positions $m=4$, $n=2$, predict which lane turns negative before looking at the figure. The pair scores are $\cos(-2\theta_i)$:

Complex multiplication is the same 2D rotation: $(x + iy)\,e^{i m \theta}$. The next helper rotates a whole head, then checks the shared-shift property across all its pairs.

import cmath
import math

def rope_freqs(dim: int, base: float = 10000.0) -> list[float]:
    if type(dim) is not int or dim <= 0 or dim % 2 or not math.isfinite(base) or base <= 0:
        raise ValueError("rotary dimension must be positive and even; base must be positive")
    return [1.0 / (base ** (index / dim)) for index in range(0, dim, 2)]

def rotate(vec: list[float], pos: int, freqs: list[float]) -> list[float]:
    if len(vec) != 2 * len(freqs):
        raise ValueError("this helper rotates the full vector: one frequency per pair")
    rotated: list[float] = []
    for pair, freq in enumerate(freqs):
        x, y = vec[2 * pair], vec[2 * pair + 1]
        spun = complex(x, y) * cmath.rect(1.0, pos * freq)
        rotated.extend([spun.real, spun.imag])
    return rotated

def dot(left: list[float], right: list[float]) -> float:
    return sum(x * y for x, y in zip(left, right))

freqs = rope_freqs(dim=4)
vec = [1.0, 0.0, 1.0, 0.0]
score_a = dot(rotate(vec, 0, freqs), rotate(vec, 2, freqs))
score_b = dot(rotate(vec, 1, freqs), rotate(vec, 3, freqs))
print(round(score_a, 6), round(score_b, 6))
print("norm preserved:", abs(dot(vec, vec) - dot(rotate(vec, 2, freqs), rotate(vec, 2, freqs))) < 1e-12)

0.583653 0.583653
norm preserved: True

This helper uses adjacent pairs and rotates the entire vector. With partial RoPE, rotate only the configured prefix and concatenate the untouched suffix. Other implementations pair the first and second halves instead. These conventions are equivalent after a matching coordinate permutation, but swapping them on an existing checkpoint without converting its projections changes its scores.^{[9]Reference 9Llama rotary embedding implementationhttps://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py}

For four coordinates, adjacent pairing reads [(x0, y0), (x1, y1)] from [x0, y0, x1, y1]. Split-half rotate_half reads the same pairs from [x0, x1, y0, y1]: its partner transform is [-y0, -y1, x0, x1]. The cosine/sine arrays must follow that layout too. Meta's reference uses adjacent complex pairs; Transformers' Llama code uses split halves.^{[8]Reference 8Llama 3.1 reference rotary embedding and frequency scalinghttps://github.com/meta-llama/llama-models/blob/f45cdfd624b98b6655540f7101d8d9cb432e631c/models/llama3_1/reference_impl/model.py[9]Reference 9Llama rotary embedding implementationhttps://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py}

Both rotations preserve norms, so a norm check won't catch a layout mismatch. Compare scores after converting both Q and K, then deliberately skip the conversion:

import math

def rotate_layout(vector, position, frequencies, split_half=False):
    result = vector.copy()
    half = len(vector) // 2
    for pair, frequency in enumerate(frequencies):
        a, b = (pair, pair + half) if split_half else (2 * pair, 2 * pair + 1)
        cosine, sine = math.cos(position * frequency), math.sin(position * frequency)
        result[a] = vector[a] * cosine - vector[b] * sine
        result[b] = vector[a] * sine + vector[b] * cosine
    return result

def score(q, k, split_half=False):
    rq = rotate_layout(q, 5, [1.0, 0.01], split_half)
    rk = rotate_layout(k, 2, [1.0, 0.01], split_half)
    return sum(a * b for a, b in zip(rq, rk))

q, k = [1.0, 2.0, 3.0, 4.0], [4.0, 3.0, 2.0, 1.0]
permutation = [0, 2, 1, 3]
converted_q = [q[i] for i in permutation]
converted_k = [k[i] for i in permutation]
adjacent = score(q, k)
converted = score(converted_q, converted_k, split_half=True)
wrong_layout = score(q, k, split_half=True)

print("adjacent score", round(adjacent, 6))
print("converted split-half score", round(converted, 6))
print("unconverted split-half score", round(wrong_layout, 6))
assert math.isclose(adjacent, converted, abs_tol=1e-12)
assert not math.isclose(adjacent, wrong_layout, abs_tol=1e-12)

adjacent score -0.760002
converted split-half score -0.760002
unconverted split-half score -1.61558

For a checkpoint conversion, this permutation belongs on the Q/K projection output coordinates for every affected head, including their biases if present. Keep it within the configured rotary region for partial RoPE. Reordering cached rotated keys or swapping a helper alone doesn't convert the model.

RoPE rotates Q and K, not V in the standard attention setup here. Q and K route attention, so their relative geometry decides which value rows get weight. V supplies the rows being mixed and isn’t directly rotated in that attention step, though earlier layers can already have put position-dependent context into it.

Keep routing and aggregation separate when comparing the three methods:

Image patches need coordinates, not just storage slots

The vision lesson flattened a patch grid into rows. That storage order doesn't require a position method to forget the grid. A simple axial RoPE assigns one set of coordinate pairs to height and another to width. With just two pairs, write

Its QK identity now depends on two offsets, $(h_k-h_q,w_k-w_q)$, rather than one sequence distance. Shifting both patches by the same height and width leaves the fixed-vector score unchanged. Raster index h * grid_width + w has a different limitation: the index gap for vertically adjacent patches changes when the grid width changes.

Qwen2.5-VL's report describes a three-axis MRoPE design for time, height, and width. Text uses the same position ID on all axes; an image uses a constant temporal ID and spatial IDs for its patch locations.^{[10]Reference 10Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923} The next two-axis toy isolates that geometry. It isn't the checkpoint's complete frequency allocation or processor.

import math

def rotate_patch(vector, height, width):
    result = []
    for (x, y), coordinate in zip([vector[:2], vector[2:]], [height, width]):
        angle = 0.5 * coordinate
        result.extend([x * math.cos(angle) - y * math.sin(angle),
                       x * math.sin(angle) + y * math.cos(angle)])
    return result

def patch_score(query_position, key_position):
    vector = [1.0, 0.0, 1.0, 0.0]
    query = rotate_patch(vector, *query_position)
    key = rotate_patch(vector, *key_position)
    return sum(q * k for q, k in zip(query, key))

original = patch_score((1, 1), (0, 1))
shifted = patch_score((5, 8), (4, 8))
lost_height = patch_score((0, 1), (0, 1))
print("one row apart", round(original, 6))
print("shared spatial shift", round(shifted, 6))
print("height IDs discarded", round(lost_height, 6))
vertical_gaps = []
for grid_width in [2, 4]:
    query_index = 1 * grid_width + 1  # query at (height=1, width=1)
    key_index = 0 * grid_width + 1    # key directly above it
    vertical_gaps.append(query_index - key_index)
print("vertical raster gaps", vertical_gaps)
assert math.isclose(original, shifted, abs_tol=1e-12)
assert not math.isclose(original, lost_height, abs_tol=1e-12)

one row apart 1.877583
shared spatial shift 1.877583
height IDs discarded 2.0
vertical raster gaps [2, 4]

Dropping height IDs makes the two different patches appear colocated to this position transform. In a multimodal runtime, preserve the processor's axis IDs, frequency-band allocation, and the checkpoint's continuation-position rule. Advancing every axis by the cache's token count invents a different geometry.^{[11]Reference 11Utilities for Rotary Embeddinghttps://huggingface.co/docs/transformers/main/internal/rope_utils}

A relative phase can rise again

RoFormer analyzes a long-term decay bound aggregated over frequencies, not a guarantee that one pair’s cosine falls forever.^{[5]Reference 5RoFormer: Enhanced Transformer with Rotary Position Embedding.https://arxiv.org/abs/2104.09864} For identical unit vectors in one pair, $S_{\Delta} = \cos(\Delta\theta)$, which oscillates.

That distinction is worth testing directly. If distance itself must always lower the positional term, what should happen when a cosine comes back up?

import math

theta = 0.5
distances = [1, 2, 4, 6, 8]
scores = [math.cos(distance * theta) for distance in distances]
print(distances)
print([round(score, 3) for score in scores])
print(scores[-1] > scores[-2])

[1, 2, 4, 6, 8]
[0.878, 0.54, -0.416, -0.99, -0.654]
True

The non-monotonic example is not just a toy warning. Men et al. show a production trap in Llama-2 experiments: perplexity can look acceptable while retrieval collapses. They saw this with a RoPE base below their derived lower bound, including a small base such as 500, and in some larger-base settings where next-token loss stayed low.^{[12]Reference 12Base of RoPE Bounds Context Length.https://arxiv.org/abs/2405.14591} A changed RoPE config still needs retrieval and task evaluation at the deployed length.

RoPE exposes offset through phase. ALiBi makes a different choice: it writes “farther is weaker” directly into the score.

ALiBi: make recency a score bias

Suppose two visible keys have the exact same raw dot-product score with a query, but one sits four positions earlier in the sequence. Should an attention head be allowed to prefer the more recent key by a fixed margin? ALiBi (Attention with Linear Biases) answers yes without altering token embeddings or rotating queries and keys. Instead of modifying vectors, it injects a static, head-specific linear distance penalty directly into the pre-softmax attention logits:^{[7]Reference 7Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.https://arxiv.org/abs/2108.12409}

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + B^{(h)}\right) V,\quad B^{(h)}_{ij} = -m_h(i-j)\ \text{for}\ j \le i$

In a causal decoder, the current token at index $i$ suffers zero penalty when attending to itself ($i - j = 0$). One step back receives $-m_h$, two steps back receives $-2m_h$, and so on. Future keys ($j > i$) are hard-masked with $-\infty$ by the causal mask. Notice an operational rule: add the bias after dividing $QK^T$ by $\sqrt{d_k}$. Dividing the bias by $\sqrt{d_k}$ would silently weaken every slope and tie the recency penalty to the per-head dimension.

Slope schedule derivation and the recency spectrum

The head slopes $m_h$ control the strength of each head's recency bias. For a power-of-two head count $H$, the reference slopes form a geometric progression with ratio $2^{-8/H}$:

$m_h = \left(2^{-8/H}\right)^h = 2^{-8h/H}, \quad h \in \{1, 2, \dots, H\}$

The smallest slope is $2^{-8}=1/256$ in this schedule. The largest is $2^{-8/H}$ and therefore changes with head count: $1/4$ for four heads, $1/2$ for eight, and $1/\sqrt{2}$ for sixteen. For the eight-head case:

• Head 1 ($m=1/2$): Ten positions back gives a $-5$ bias. Against a self-key with equal content score, its softmax odds ratio is $e^{-5}\approx0.0067$. That is not a universal multiplier on its normalized probability, and it does not make a hard window.
• Head 8 ($m=1/256$): At distance 256, the bias is $-1$. It applies a weaker penalty than Head 1; actual attention still depends on content and the other visible keys.

For 16 heads, the ratio is $2^{-8/16} = 2^{-0.5} = 1/\sqrt{2} \approx 0.7071$, yielding slopes $2^{-0.5}, 2^{-1.0}, 2^{-1.5}, \dots, 2^{-8.0}$. If a model has a non-power-of-two head count $H$, the reference implementation finds the closest power of two $p = 2^{\lfloor \log_2 H \rfloor}$, constructs the $p$-head schedule, and appends the first $H - p$ slopes sampled at stride 2 from a $2p$-head schedule.^{[13]Reference 13ALiBi reference transformer implementationhttps://github.com/ofirpress/attention_with_linear_biases/blob/master/fairseq/models/transformer.py}

Read the slope table as a prediction tool: which head should hold onto an older key more strongly?

Run that rule on a token-indexed slice: "The" at 0, "model" at 1, "has" at 2, and "loaded" at 3. Suppose "loaded" has a scaled QK score of $2.5$ for both "model" and "The". Head 4 has $m=1/16=0.0625$, so the distances 2 and 3 produce logits $2.375$ and $2.3125$. Head 1 would subtract $1.0$ and $1.5$ instead. With unequal content scores, a strong semantic match can still outvote the distance penalty.

The helper below applies one slope to one visible key. Its raw_score is already $q^Tk/\sqrt{d_k}$, before adding the bias. Predict the outputs for distance four and scaled content logit 4.0.

import math

def alibi_score(raw_score: float, query_pos: int, key_pos: int, slope: float) -> float:
    if (type(query_pos) is not int or type(key_pos) is not int
            or query_pos < 0 or key_pos < 0
            or not math.isfinite(raw_score) or not math.isfinite(slope) or slope < 0):
        raise ValueError("positions must be nonnegative integers; scores and slope finite, slope nonnegative")
    if key_pos > query_pos:
        return float("-inf")  # causal mask, not the distance bias
    distance = query_pos - key_pos
    return raw_score - slope * distance

raw = 4.0
query_pos = 10
key_pos = 6
head_1 = alibi_score(raw, query_pos, key_pos, slope=1 / 2)
head_8 = alibi_score(raw, query_pos, key_pos, slope=1 / 256)
print(head_1, round(head_8, 3))
print("future:", alibi_score(raw, query_pos, key_pos=11, slope=1 / 2))

2.0 3.984
future: -inf

One score hides an important boundary, so the next snippet builds the full causal bias tensor. It includes the non-power-of-two slope helper from the reference implementation and writes $-\infty$ only for future keys.

import math

def alibi_slopes_power_of_two(n_heads: int) -> list[float]:
    if type(n_heads) is not int or n_heads <= 0 or n_heads & (n_heads - 1):
        raise ValueError("n_heads must be a positive power of two")
    start = 2 ** (-8 / n_heads)
    return [start * (start ** head) for head in range(n_heads)]

def alibi_slopes(n_heads: int) -> list[float]:
    if type(n_heads) is not int or n_heads <= 0:
        raise ValueError("n_heads must be a positive integer")
    if n_heads & (n_heads - 1) == 0:
        return alibi_slopes_power_of_two(n_heads)
    closest = 1 << (n_heads.bit_length() - 1)
    extra = alibi_slopes(2 * closest)[0::2][: n_heads - closest]
    return alibi_slopes_power_of_two(closest) + extra

def build_alibi_bias(n_heads: int, seq_len: int) -> list[list[list[float]]]:
    if type(seq_len) is not int or seq_len <= 0:
        raise ValueError("seq_len must be a positive integer")
    slopes = alibi_slopes(n_heads)
    bias: list[list[list[float]]] = []
    for slope in slopes:
        head = []
        for query in range(seq_len):
            row = []
            for key in range(seq_len):
                if key > query:
                    row.append(float("-inf"))
                else:
                    row.append(-slope * (query - key))
            head.append(row)
        bias.append(head)
    return bias

bias = build_alibi_bias(n_heads=4, seq_len=4)
print(len(bias), len(bias[0]), len(bias[0][0]))
print([round(value, 4) if math.isfinite(value) else value for value in bias[0][3]])
print([round(value, 4) if math.isfinite(value) else value for value in bias[-1][3]])
print("3-head slopes:", alibi_slopes(3))

4 4 4
[-0.75, -0.5, -0.25, -0.0]
[-0.0117, -0.0078, -0.0039, -0.0]
3-head slopes: [0.0625, 0.00390625, 0.25]

Extrapolation mechanics and the long-range penalty trap

ALiBi's rule has no finite table boundary or periodic phase. It can be evaluated at new distances; those larger biases and visibility patterns can still be unfamiliar to a checkpoint. Press et al.'s 1.3B experiment trained at length 1,024 and evaluated at 2,048, matching a longer-trained sinusoidal baseline's perplexity. They measured about 11% less memory and 11% faster progress to a given validation perplexity in that setup.^{[7]Reference 7Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.https://arxiv.org/abs/2108.12409} These are particular training comparisons, not a speed or quality guarantee for every model.

Its monotonic recency prior is a tradeoff worth calculating. At distance 16,384, the eight-head slopes above give biases of $-8192$ and $-64$ at the extremes. Compare the gentlest head's far key against the self-key: with equal content scores, the far-to-self odds are $e^{-64}$. A content-score advantage greater than 64 reverses that two-key comparison. Against an adjacent key, the bias difference is $(16384-1)/256$, not exactly 64.

Content logits depend on learned norms and projections, and other heads and layers can relay information. A single head's penalty therefore does not determine the whole model's retrieval ability. RoPE uses relative phase rather than this linear bias; its frequency and training choices also need evaluation.

import math

def softmax(logits: list[float]) -> list[float]:
    maximum = max(logits)
    shifted = [math.exp(value - maximum) for value in logits]
    total = sum(shifted)
    return [value / total for value in shifted]

raw_scores = [2.0, 2.0, 2.0, 100.0]
query_position = 2
biased_scores = [
    score - 0.5 * (query_position - key) if key <= query_position else -math.inf
    for key, score in enumerate(raw_scores)
]
weights = softmax(biased_scores)
print([round(weight, 3) for weight in weights])
print(weights[0] > 0)
print("future weight:", weights[3])
far_vs_self = softmax([65.0 - 16384 / 256, 0.0])
print("far key with a 65-logit advantage:", round(far_vs_self[0], 3))

[0.186, 0.307, 0.506, 0.0]
True
future weight: 0.0
far key with a 65-logit advantage: 0.731

A closed-form bias can be evaluated at offsets the model never trained on. Whether the checkpoint still retrieves buried evidence is a different question, and long-context evaluation makes that question concrete.

Long context: a defined formula still needs evidence

Here's the operational handoff: a checkpoint trained at 4K now has a 32K serving target, with a passkey early in the document and a retrieval question near token 30,000. We need to test the large separation between evidence and query, not merely a large absolute index. Writing down $PE_{32000}$ or $R_{32000}$ only proves the rule is defined.

Unchanged RoPE evaluates larger offsets without difficulty in real-number arithmetic. Fast phases may already have wrapped many times during training; slower bands may enter less-trained phase regions. Longer sequences also change distractor count and context-dependent features. Direct extension failed in Chen et al.'s tested LLaMA runs, but "an unseen angle destroys attention" is not a complete or universal explanation.^{[14]Reference 14Extending Context Window of Large Language Models via Positional Interpolation.https://arxiv.org/abs/2306.15595[12]Reference 12Base of RoPE Bounds Context Length.https://arxiv.org/abs/2405.14591} Compute phases accurately and measure behavior.

Use the table as an experiment map. Each row separates what arithmetic does past the training length from what a paper actually measured.

Use two evidence gates: short-context behavior must survive, and retrieval plus target tasks must work at the new length. Sweep evidence positions and evidence-to-query distances, include distractors and missing-evidence cases, and reserve tokens for generation. Average next-token perplexity can be dominated by locally predictable text; it does not replace those tests.^{[12]Reference 12Base of RoPE Bounds Context Length.https://arxiv.org/abs/2405.14591}

Even correct position IDs can be lost before the rotation. Casting indices 4096 and 4097 to bfloat16 rounds both to 4096. Computing cosine in float32 afterward can't recover the distinction. Transformers' Llama implementation computes phases from floating-point position IDs and frequencies in float32, then casts the resulting sine/cosine tensors to the activation dtype.^{[9]Reference 9Llama rotary embedding implementationhttps://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py}

import torch

positions = torch.tensor([4096, 4097], dtype=torch.int64)
rounded_first = positions.to(torch.bfloat16).float()
phase_fp32 = positions.float()  # fastest pair has frequency 1
bad_cosine = rounded_first.cos()
good_cosine = phase_fp32.cos()

print("positions after bfloat16", rounded_first.tolist())
print("bad phases identical", bool(torch.equal(rounded_first[:1], rounded_first[1:])))
print("bad cosines", [round(x, 6) for x in bad_cosine.tolist()])
print("float32 cosines", [round(x, 6) for x in good_cosine.tolist()])
assert bad_cosine[0] == bad_cosine[1]
assert good_cosine[0] != good_cosine[1]

positions after bfloat16 [4096.0, 4096.0]
bad phases identical True
bad cosines [0.803991, 0.803991]
float32 cosines [0.803991, 0.934772]

Keep position IDs as integers until deliberate phase computation. Check the phase tensor's dtype separately from Q/K and the cache dtype. This CPU example diagnoses a conversion error; it doesn't benchmark a fused kernel. float32 also has finite precision, so a proposed much larger range needs numerical checks at its actual indices.

PI: compress coordinates into the trained range

Position Interpolation (PI) compresses the target sequence positions so they fit within the pretraining domain:^{[14]Reference 14Extending Context Window of Large Language Models via Positional Interpolation.https://arxiv.org/abs/2306.15595}

$m' = m \cdot \frac{L_{\text{train}}}{L_{\text{target}}} = \frac{m}{s}$

where $s=L_{\text{target}}/L_{\text{train}}$. This is equivalent to using frequencies $\theta_i/s$ with unchanged positions. For target indices $0\le m<L_{\text{target}}$, effective coordinates satisfy $0\le m'<L_{\text{train}}$. They can be fractional; PI is not a lookup of an unchanged integer position table.

Chen et al. extended tested 2,048-token LLaMA checkpoints up to 32,768 with fine-tuning within 1,000 steps.^{[14]Reference 14Extending Context Window of Large Language Models via Positional Interpolation.https://arxiv.org/abs/2306.15595} For our separate scale-4 example, position 16,383 maps to $4095.75$. This is below 4,096 but slightly above the last integer training index 4,095. "Compressed into the window" does not mean every coordinate was observed exactly during pretraining.

train_length = 4096
target_length = 16384
scale = train_length / target_length

for target_position in [0, 4096, 8192, 16383]:
    effective_position = target_position * scale
    print(target_position, round(effective_position, 2))

0 0.0
4096 1024.0
8192 2048.0
16383 4095.75

PI also compresses every adjacent-token phase difference by $1/s$, including the fastest pair. YaRN's authors hypothesize that this loss of high-frequency separation contributes to short-context degradation in tested extensions.^{[15]Reference 15YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071} It does not make adjacent indices identical or prove a universal grammar failure. Measure short tasks after adaptation as well as long retrieval.

NTK-aware scaling: stretch frequencies unevenly

The name NTK-aware comes from reasoning about neural tangent kernels and Fourier-feature frequency separation. It is not a theorem guaranteeing length generalization for a trained LLM. The proposed base transform stretches lower frequencies more while preserving the fastest pair:^{[15]Reference 15YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071}

$b' = b \cdot s^{d/(d-2)}$

Here $d=d_{\text{rotary}}>2$ is even. The formula is undefined at $d=2$. Consider its effect on $\theta_i'=(b')^{-2i/d}$:

$\theta_i' = \left( b \cdot s^{d/(d-2)} \right)^{-2i/d} = b^{-2i/d} \cdot s^{-2i/(d-2)} = \theta_i \cdot s^{-2i/(d-2)}$

Analyze the two extremes of that spectrum:

• Highest frequency ($i=0$): $\theta_0'=1$. This pair keeps its original phase step; it does not guarantee preserved syntax or all local model behavior.
• Lowest frequency ($i=d/2-1$): $\theta_{d/2-1}'=\theta_{d/2-1}/s$.
• Middle frequencies: Smoothly transition from zero compression to full interpolation.

Code Llama's paper raises its base to $10^6$ during long-context fine-tuning.^{[16]Reference 16Code Llama: Open Foundation Models for Codehttps://arxiv.org/abs/2308.12950} A manually chosen larger base is related to this frequency-stretching idea; it does not by itself establish that the checkpoint uses the exact scale formula above.

YaRN: treat frequency bands differently

YaRN combines NTK-by-parts frequency treatment with an empirical attention-temperature adjustment.^{[15]Reference 15YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071} Start with each pair's wavelength $\lambda_i=2\pi/\theta_i$ and count rotations across the original window:

$r_i=\frac{L_{\text{train}}}{\lambda_i}=\frac{L_{\text{train}}\theta_i}{2\pi}.$

The paper's thresholds $\alpha$ and $\beta$ are rotation counts, not token wavelengths. Its illustrative LLaMA settings use $\alpha=1$ and $\beta=32$. Set

$\gamma_i=\operatorname{clip}\left(\frac{r_i-\alpha}{\beta-\alpha},0,1\right),\qquad \theta_i'=\left(1-\gamma_i\right)\frac{\theta_i}{s}+\gamma_i\theta_i.$

Pairs completing fewer than one turn use full interpolation; those completing more than 32 keep their frequencies. The middle blends them. Transformers 5.17.0's YaRN code instead ramps linearly over pair indices between correction bounds, with optional bound rounding.^{[17]Reference 17Transformers 5.17.0 YaRN rotary parameter implementationhttps://github.com/huggingface/transformers/blob/v5.17.0/src/transformers/modeling_rope_utils.py} The helper below implements the paper's rotation-count ramp, not that runtime's exact output. Match the checkpoint's implementation; neither ramp guarantees local syntax.

Try $d=8$, $L=4096$, and $s=4$. The four original frequencies are $[1,0.1,0.01,0.001]$. Which two complete more than 32 turns? Which completes less than one?

import math

def yarn_paper_frequencies(dim: int, train_length: int, scale: float,
                           base: float = 10000.0,
                           alpha: float = 1.0, beta: float = 32.0) -> list[float]:
    if (type(dim) is not int or dim <= 0 or dim % 2
            or type(train_length) is not int or train_length <= 0
            or not all(math.isfinite(x) for x in (scale, base, alpha, beta))
            or scale < 1 or base <= 1 or not 0 <= alpha < beta):
        raise ValueError("positive even dim, positive integer length, finite scale >= 1, base > 1, 0 <= alpha < beta")
    frequencies = [base ** (-2 * pair / dim) for pair in range(dim // 2)]
    result = []
    for frequency in frequencies:
        turns = train_length * frequency / (2 * math.pi)
        preserve = min(1.0, max(0.0, (turns - alpha) / (beta - alpha)))
        result.append(frequency * (preserve + (1 - preserve) / scale))
    return result

original = [1.0, 0.1, 0.01, 0.001]
adjusted = yarn_paper_frequencies(8, 4096, 4)
turns = [4096 * frequency / (2 * math.pi) for frequency in original]
print("turns:", [round(value, 3) for value in turns])
print("new frequencies:", [round(value, 6) for value in adjusted])
print("ratios:", [round(new / old, 3) for new, old in zip(adjusted, original)])

turns: [651.899, 65.19, 6.519, 0.652]
new frequencies: [1.0, 0.1, 0.003835, 0.00025]
ratios: [1.0, 1.0, 0.384, 0.25]

YaRN additionally uses $a=\sqrt{1/t}=1+0.1\ln s$ as an empirical fit for tested LLaMA/Llama 2 models. Multiplying both rotated Q and K by $a$ multiplies their dot-product logits by $a^2$. For $s=4$, $a\approx1.139$, $a^2\approx1.296$, and $t\approx0.771$. This scaling changes norms and, for nonuniform logits, sharpens softmax. More keys do not inevitably increase entropy: a newly added key with a dominant score can concentrate the distribution. The empirical adjustment needs its own task evaluation.

import math

rotary_dim, base, scale = 128, 10000.0, 4.0
new_base = base * scale ** (rotary_dim / (rotary_dim - 2))
ratios = [(base / new_base) ** (2 * pair / rotary_dim) for pair in [0, 16, 32, 48, 63]]
amplitude = 1 + 0.1 * math.log(scale)
print("new base:", round(new_base))
print("frequency ratios:", [round(value, 3) for value in ratios])
print("q/k factor:", round(amplitude, 3))
print("logit factor:", round(amplitude ** 2, 3))
print("temperature:", round(1 / amplitude ** 2, 3))

new base: 40890
frequency ratios: [1.0, 0.703, 0.495, 0.348, 0.25]
q/k factor: 1.139
logit factor: 1.296
temperature: 0.771

Cache positions are part of the computation

During cached generation, use the new query's actual sequence position and the same frequency schedule as its cached keys. A cache's storage-slot number is not necessarily its position ID, especially with padding, packed examples, or sliding retention. Restarting the query at zero changes offsets. Rotating a key twice changes them too.

For one fixed-Q/K layer, the next CPU example compares a full causal pass with token-by-token cached attention. It then resets one query's position and changes only the query's frequency schedule. Predict which comparisons survive.

import torch
import torch.nn.functional as F

def rotate_pairs(rows: torch.Tensor, positions: torch.Tensor,
                 frequencies: torch.Tensor) -> torch.Tensor:
    if rows.ndim != 2 or rows.shape[1] != 2 * frequencies.numel():
        raise ValueError("one frequency per adjacent coordinate pair")
    if positions.shape != (rows.shape[0],) or frequencies.ndim != 1:
        raise ValueError("one position per row and a one-dimensional frequency schedule")
    pairs = rows.reshape(rows.shape[0], -1, 2)
    angles = positions[:, None] * frequencies[None, :]
    cos, sin = angles.cos(), angles.sin()
    x, y = pairs[..., 0], pairs[..., 1]
    return torch.stack((x * cos - y * sin, x * sin + y * cos), -1).flatten(1)

q = torch.tensor([[1., 0., 1., 0.], [0., 1., 1., 1.],
                  [1., 1., 0., 1.]], dtype=torch.float64)
k = torch.tensor([[1., 1., 1., 0.], [1., 0., 0., 1.],
                  [0., 1., 1., 1.]], dtype=torch.float64)
v = torch.eye(3, dtype=torch.float64)
positions = torch.arange(800, 803, dtype=torch.float64)
frequencies = torch.tensor([1., 0.01], dtype=torch.float64)
rotated_q = rotate_pairs(q, positions, frequencies)
rotated_k = rotate_pairs(k, positions, frequencies)

def attend(query, keys, values, causal=False, mask=None):
    return F.scaled_dot_product_attention(query[None, None], keys[None, None],
                                         values[None, None], dropout_p=0.,
                                         is_causal=causal, attn_mask=mask)[0, 0]

full = attend(rotated_q, rotated_k, v, causal=True)
cached = torch.cat([attend(rotated_q[i:i+1], rotated_k[:i+1], v[:i+1])
                    for i in range(3)])
chunk_mask = positions[None, :] <= positions[1:, None]
chunk = attend(rotated_q[1:], rotated_k, v, mask=chunk_mask)
unmasked_chunk = attend(rotated_q[1:], rotated_k, v)
reset_query = rotate_pairs(q[-1:], torch.zeros(1, dtype=q.dtype), frequencies)
new_frequencies = torch.tensor([1., 0.005], dtype=q.dtype)
mixed_query = rotate_pairs(q[-1:], positions[-1:], new_frequencies)
consistent_keys = rotate_pairs(k, positions, new_frequencies)
fixed_layer_repaired = attend(mixed_query, consistent_keys, v)
all_new = attend(rotate_pairs(q, positions, new_frequencies), consistent_keys, v, causal=True)

print("full and cached match:", torch.allclose(full, cached, atol=1e-12, rtol=0))
print("full and cached chunk match:", torch.allclose(full[1:], chunk, atol=1e-12, rtol=0))
print("chunk without mask matches:", torch.allclose(full[1:], unmasked_chunk, atol=1e-12, rtol=0))
print("chunk first query sees future:", bool(chunk_mask[0, -1]))
print("reset position matches:", torch.allclose(full[-1:], attend(reset_query, rotated_k, v), atol=1e-12, rtol=0))
print("mixed schedules match new pass:", torch.allclose(all_new[-1:], attend(mixed_query, rotated_k, v), atol=1e-12, rtol=0))
print("fixed layer repaired:", torch.allclose(all_new[-1:], fixed_layer_repaired, atol=1e-12, rtol=0))

full and cached match: True
full and cached chunk match: True
chunk without mask matches: False
chunk first query sees future: False
reset position matches: False
mixed schedules match new pass: False
fixed layer repaired: True

Every single-token step sees only its current prefix, so it needs no additional causal mask. Its one-row query is already the last position. The two-token chunk is different: query 801 must hide key 802, while query 802 may see it. The explicit mask compares their logical positions; in this PyTorch API, True permits a pair. A rectangular is_causal=True call can have different alignment; use the explicit offset-aware mask from the attention lesson for cached chunks. Matching rotary offsets doesn't repair future-token leakage.

YaRN's paper recommends caching unrotated keys when applying a changing schedule.^{[15]Reference 15YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071} Converting old rotated keys with $R_{\mathrm{new}}(p)R_{\mathrm{old}}(p)^T$ repairs the rotation for fixed unrotated keys in exact arithmetic. In a multilayer decoder, earlier attention changes can also change later K/V features; rerotating stored keys alone need not reproduce a full pass under the new schedule. Use a supported cache policy or recompute the prefix when full-pass equivalence is required. This toy verifies arithmetic, not trained-model cache quality.

Read the whole checkpoint configuration

Llama 3.1's reference code starts from base $500{,}000$ and uses a piecewise transform based on the original 8,192-token window.^{[8]Reference 8Llama 3.1 reference rotary embedding and frequency scalinghttps://github.com/meta-llama/llama-models/blob/f45cdfd624b98b6655540f7101d8d9cb432e631c/models/llama3_1/reference_impl/model.py} Wavelengths below 2,048 stay unchanged; those above 8,192 have frequencies divided by eight; the middle blends the two using the rotation-count ratio. The factor eight is a frequency-scaling parameter, not an arithmetic proof of the checkpoint's 128K window. The Llama 3 report also describes staged long-context training.^{[6]Reference 6The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Gemma 3 (2025) uses five local attention layers for each global layer, with bases 10,000 locally and 1,000,000 globally.^{[18]Reference 18Gemma 3 Technical Reporthttps://arxiv.org/abs/2503.19786} Its report combines global RoPE rescaling with a final long-context training stage for the 4B/12B/27B models. One base number does not describe that whole design.

LongRoPE2 (2025) searches per-dimension scaling using loss on synthetic needle-answer tokens, then mixes original-RoPE short-context training and rescaled-RoPE long-context training.^{[19]Reference 19LongRoPE2: Near-Lossless LLM Context Window Scalinghttps://arxiv.org/abs/2502.20082} Its LLaMA3-8B and Phi3-mini experiments illustrate a useful principle: optimize and measure distant-evidence use, while checking short-context regression. The title's "near-lossless" describes the paper's evaluated results, not every downstream task.

As checked October 10, 2026, Transformers' documentation exposes variants through rope_parameters and supports different settings per layer type.^{[11]Reference 11Utilities for Rotary Embeddinghttps://huggingface.co/docs/transformers/main/internal/rope_utils} Load the checkpoint's settings and match the installed runtime's schema. The same advertised method name can hide differences in rotary width, pairing, base, frequency ramp, amplitude, and cache policy.

Choose by mechanism, then earn the window

Choose the insertion point that matches the behavior you want. Then test short-context quality, long-range retrieval, generation, and target tasks at the length you plan to advertise.

The implementation choice decides where position enters attention. The checkpoint still has to earn the window you advertise.

Check the three insertion points

• Explain why unmasked self-attention without a position signal is permutation equivariant.
• Point to where sinusoids, RoPE, and ALiBi insert position: embeddings, Q/K geometry, or logits.
• Compute a RoPE relative phase and an ALiBi distance penalty on a tiny example.
• Compare full-pass and cached-chunk outputs while holding pairing, frequency schedule, position IDs, and permitted keys fixed.
• Treat configured context length as a claim that needs retrieval and task evidence at the deployed length.

Common misconceptions

• Order seems implicit: Symptom: you say unmasked attention already knows sequence order. Cause: you skipped the permutation-equivariance check. Fix: start from content-only attention, then account separately for position signals and masking.
• All methods sound the same: Symptom: sinusoids, learned tables, RoPE, and ALiBi blur together. Cause: you aren't tracking the insertion point. Fix: embeddings vs Q/K geometry vs logit bias.
• RoPE rotates the wrong tensor: Symptom: you apply RoPE to embeddings or V. Cause: routing and aggregation got merged. Fix: rotate Q and K after projection; leave V without a direct RoPE rotation.
• Long context looks solved on paper: Symptom: perplexity improves but retrieval still fails. Cause: you treated mathematical validity as behavioral proof. Fix: run passkey, retrieval, summarization, and target-task evals at the deployed length.
• Any base change is enough: Symptom: the config advertises a larger window but distant facts disappear. Cause: a new $\theta$ doesn't teach the model to use long-range evidence. Fix: pair geometry changes with an adaptation method, long-context data, and task-level checks.
• Concatenate by default: Symptom: width and projection cost balloon. Cause: you doubled the hidden width without accounting for downstream matrices. Fix: reproduce the checkpoint's position design; a new concatenated design can project back down or explicitly budget for a wider stack.