Positional Encoding: RoPE & ALiBi

Put two lines from a deployment log on a screen: “The model loaded the checkpoint” and “The checkpoint loaded the model.” They reuse the same words, but report opposite events. A model that only compares token content has no slot index to tell those commands apart.

Vision transformers expose that gap with image patches, and text has it too. The self-attention mechanism compares content, not position. Predict what happens when the token rows for “model,” “loaded,” and “ckpt” are shuffled.

The answer gives us our first diagnostic: unmasked attention shuffles its outputs along with its inputs. That’s permutation equivariance, and it explains why transformers need a position signal. We’ll follow the signal through three insertion points: vectors added to embeddings, RoPE rotations on queries and keys, and ALiBi’s linear distance bias on attention logits.

Then we’ll stress the same idea at long length. A formula can produce a value at position 32,000 even when training stopped at 4,096. That calculation is useful, but it isn’t evidence that the model can find a buried instruction there.

One boundary stays fixed throughout: a position method can rank permitted keys, while a causal mask must still hide future keys. A useful test changes order, checks the permitted scores, and confirms that forbidden positions remain forbidden.

---

First, watch attention lose the order

Before adding any position feature, make one prediction. If every token vector stays the same but the rows move, should content-only attention preserve the original row order, or should its outputs move with them?

Self-attention computes:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

The product compares every query with every key ($QK^T$), scales by $\sqrt{d_k}$ (the key width), turns scores into softmax weights, and mixes value rows $V$. No term reads “first,” “second,” or “third.” The weights depend on content, not index.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}

So a row permutation has a predictable effect: it also permutes Q, K, and V. Every query sees the same collection of content vectors, only in a new order, and each output row follows its query.

Use a three-token slice of our running example, with one 2D vector per token. Predict the shuffled output before checking the table:

Permutation $P = [2, 0, 1]$ rewrites the sequence as ckpt, model, loaded. Unmasked attention on the shuffled rows is exactly the shuffled original output: $Y' = P\,Y$. The block has preserved content relationships, not command order.

The small check below recomputes attention after the permutation. Read True as a diagnosis: no hidden slot number entered the calculation.

A causal mask changes this result by removing future keys. A decoder-only model can then infer some order from which positions are visible, and Haviv et al. found NoPos runs competitive with sinusoidal and learned embeddings for causal language modeling.^{[2]Reference 2Transformer Language Models without Positional Encodings Still Learn Positional Information.https://arxiv.org/abs/2203.16634} Their bidirectional masked models failed to converge without an explicit position signal. Visibility gives a direction, but it still doesn’t provide a numeric coordinate or distance feature.

That distinction tells us where to look next. Absolute encodings, RoPE, and ALiBi answer the same ordering gap at different points in the block. They’re alternatives, not stacked stages.

The table is a map of those insertion points. For each row, ask what downstream quantity can first “feel” position.

When debugging a position mismatch, inspect the first tensor that changed: the embedding, projected Q/K, or attention score matrix. That check also catches accidental stacking of methods meant to be alternatives.

The first map entry adds position before any Q, K, or V projection. Start there, then we’ll move the signal deeper into attention.

---

Absolute position: give every slot a multi-speed fingerprint

Nearby slots need distinguishable labels, while far slots need a slower signal that tracks broad location. The original Transformer meets both needs with several paired waves added to each token embedding.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}

At position 0, every sine starts at 0 and every cosine at 1. Move to position 2 and the fastest pair has moved two radians, while slower pairs barely move. The combined phases form a position fingerprint before the network sees Q, K, or V.

For position $\text{pos}$ and dimension $i$, with $d_{\text{model}}$ the embedding width:

$PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

$PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

The paired waves carry one useful algebraic property. A fixed offset is a linear transform, $PE_{\text{pos}+k} = R_k \cdot PE_{\text{pos}}$, where the rotation $R_k$ depends only on $k$. In principle, later layers can learn relative attention from that structure.

The generator below makes the dial picture concrete with an 8-by-8 table. Before running it, predict row 0 and the first pair at row 2.

The dials have different wavelengths. With $d_{\text{model}}=512$, the table below samples one fast, one middle, and one slow pair using $\lambda = 2\pi \cdot 10000^{i/d_{\text{model}}}$ for each pair’s first dimension. Use it to predict which lane changes most between neighboring tokens.

The waves are easier to read as lanes than as a matrix:

Sinusoids are fixed. BERT (Bidirectional Encoder Representations from Transformers) and GPT-2 instead learn a table $P \in \mathbb{R}^{L_{\max} \times d}$: one trainable vector per slot, hard-limited at $L_{\max}$.^{[3]Reference 3BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805[4]Reference 4GPT-2 Source Implementation.https://github.com/openai/gpt-2/blob/master/src/model.py} Predict the consequence first: what happens at a position beyond the last row?

Vaswani et al. tested both and reported “nearly identical results” (Table 3, row E). Sinusoids were attractive because their formula can be evaluated at unseen indices. That’s a definition, not a quality guarantee: Press et al. measured sharp perplexity degradation when sinusoidal models ran far past their training length on WikiText-103.^{[7]Reference 7Train Short, Test Long: Attention with Linear Biases Enables Input Length Generalization.https://arxiv.org/abs/2108.12409}

Why add instead of concatenate? Put two $d$-wide vectors side by side and the next layer sees width $2d$. Unless you project back down, square matrices such as $W_Q$, $W_K$, and $W_V$ then grow about 4x. Addition keeps width $d$, but it also gives later projections one mixed vector rather than separate content and position channels.

Why do absolute positional encodings get added to token embeddings instead of concatenated?

Answer

Concatenation would double the hidden width from $d$ to $2d$ unless the model projected back down. If the transformer stayed at width $2d$, square projections such as $W_Q$, $W_K$, and $W_V$ would become roughly 4x larger. Addition keeps width $d$ and gives later layers one combined vector. Whether those layers use the mixture well is empirical.

Absolute methods leave us with three questions. Does quality hold beyond the trained length? Can the model infer an offset from two absolute vectors? Is mixing content and position in one stream a good trade? Relative methods move the distance signal into attention itself, which is the next experiment.

---

RoPE: make a shared offset visible inside QK

Now move the position signal past the embedding. Keep one query and one key fixed, then place both at positions $(m,n)$. If both positions shift by 100, should their relative score change?

RoPE (Rotary Position Embeddings) answers by rotating each query and key in 2D subspaces. It comes from RoFormer and is used in Llama-family models, including Llama 3 with RoPE base $\theta = 500{,}000$.^{[5]Reference 5RoFormer: Enhanced Transformer with Rotary Position Embedding.https://arxiv.org/abs/2104.09864[6]Reference 6The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Hold the unrotated vectors fixed, and the dot product depends on the offset between them rather than on their shared absolute shift.

The picture below makes that prediction visible. The absolute angles move, but the phase gap doesn’t.

For one 2D pair $(q_{2i}, q_{2i+1})$ at position $m$, RoPE applies this rotation:

$$\begin{pmatrix} q_{2i}^{(m)} \\ q_{2i+1}^{(m)} \end{pmatrix} = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix} \begin{pmatrix} q_{2i} \\ q_{2i+1} \end{pmatrix}$$

Each consecutive pair spins like one clock hand. Its angle depends on the token index $m$ and the pair frequency $\theta_i$. Production RoPE usually sets those frequencies from the per-head rotary width, not the full hidden size:

$\theta_i = 10000^{-2i/d_{\text{rotary}}}$

If every head dimension is rotated, $d_{\text{rotary}} = d_k$. Some stacks rotate only part of each head, so use the configured rotary dimension when reproducing a checkpoint.

The cancellation comes from composing rotations. For one pair,

$q^{(m)\top} k^{(n)} = q^{\top} R((n-m)\theta_i)\, k$

The same unrotated vectors at $(m,n)=(5,2)$ and $(105,102)$ therefore get the same positional score. The absolute coordinates differ; the offset doesn’t.

The output confirms the shared-shift prediction. It also marks the limit of this toy proof: a trained model’s context window depends on content-dependent Q and K vectors across many layers, not on this identity alone.

Use a unit-vector walk-through to isolate the phase term. Let $q = k = [1, 0]$ and $\theta = 0.5$.

• Query at $m=2$, key at $n=1$: offset $d = n-m = -1$, score $\cos(-0.5) \approx 0.878$.
• Query at $m=102$, key at $n=101$: same offset, same score.

At offset $-2$, the same pair scores $\cos(-1.0) \approx 0.540$. Don’t turn that comparison into “farther is always weaker.” Cosine phases can rise again, and a real head sums many frequencies.

One head with $d_{\text{rotary}}=8$ contains four such pairs. With the standard base-$10000$ schedule, their frequencies are $1$, $0.1$, $0.01$, and $0.001$. For $q=k=[1,0]$ in every pair and positions $m=4$, $n=2$, predict which lane turns negative before looking at the figure. The pair scores are $\cos(-2\theta_i)$:

Complex multiplication is the same 2D rotation: $(x + iy)\,e^{i m \theta}$. The next helper rotates a whole head, then checks the shared-shift property across all its pairs.

RoPE rotates Q and K, not V. Q and K route attention, so their relative geometry decides which value rows get weight. V supplies the rows being mixed and isn’t directly rotated in that attention step, though earlier layers can already have put position-dependent context into it.

Why does RoPE rotate Q and K without directly rotating V?

Answer

Q and K decide which values receive attention weight, so rotating them makes the routing score depend on relative offset. V supplies the information being mixed and isn't rotated in that attention step. After earlier layers, V can still carry position-dependent context.

Keep routing and aggregation separate when comparing the three methods:

RoFormer analyzes a long-term decay bound aggregated over frequencies, not a guarantee that one pair’s cosine falls forever.^{[5]Reference 5RoFormer: Enhanced Transformer with Rotary Position Embedding.https://arxiv.org/abs/2104.09864} For identical unit vectors in one pair, $S_{\Delta} = \cos(\Delta\theta)$, which oscillates.

That distinction is worth testing directly. If distance itself must always lower the positional term, what should happen when a cosine comes back up?

The non-monotonic example is not just a toy warning. Men et al. show a production trap in Llama-2 experiments: perplexity can look acceptable while retrieval collapses. They saw this with a RoPE base below their derived lower bound, including a small base such as 500, and in some larger-base settings where next-token loss stayed low.^{[8]Reference 8Base of RoPE Bounds Context Length.https://arxiv.org/abs/2405.14591} A changed RoPE config still needs retrieval and task evaluation at the deployed length.

RoPE exposes offset through phase. ALiBi makes a different choice: it writes “farther is weaker” directly into the score.

---

ALiBi: make recency a score bias

Suppose two visible keys have the same raw QK score, but one sits four positions behind the query. Should a head be allowed to prefer the newer key by a fixed amount? ALiBi (Attention with Linear Biases) answers yes without changing the embedding or rotating Q and K. After the query-key product, it adds a head-specific linear bias:^{[7]Reference 7Train Short, Test Long: Attention with Linear Biases Enables Input Length Generalization.https://arxiv.org/abs/2108.12409}

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + B^{(h)}\right) V,\quad B^{(h)}_{ij} = -m_h(i-j)\ \text{for}\ j \le i$

In a causal decoder, the current token gets zero penalty, one step back gets $-m_h$, two steps back gets $-2m_h$, and so on. Future keys stay at $-\infty$ from the causal mask. Press et al. note that this bias is not multiplied by $\sqrt{d_k}$; it’s added to the already-scaled (or, in their presentation, unscaled) scores.

The head slopes supply different recency strengths. For $n$ heads, the paper starts at $2^{-8/n}$ and uses that value as the geometric ratio. For 8 heads that’s $m = \tfrac12, \tfrac14, \ldots, \tfrac1{256}$: Head 1 is short-sighted, while Head 8 barely penalizes distance. For a non-power-of-two head count, the reference implementation keeps the power-of-two schedule for the largest $p < n$, then appends every other slope from the $2p$ schedule until it has $n$ slopes.

Read the slope table as a prediction tool: which head should hold onto an older key more strongly?

Run that rule on our checkpoint example. “Loaded” at position 3 attends to “model” at position 1 with raw score $2.5$. Head 4 has $m = 1/16 = 0.0625$, so distance 2 contributes $-0.125$ and the logit becomes $2.375$. “The” at position 0 is distance 3, so its logit becomes $2.3125$. Head 1 would subtract $1.0$ and $1.5$ instead.

The helper below applies one slope to one visible key. Predict the two outputs for distance 4 and raw score 4.0 before comparing the steep and gentle heads.

One score hides an important boundary, so the next snippet builds the full causal bias tensor. It includes the non-power-of-two slope helper from the reference implementation and writes $-\infty$ only for future keys.

In one original-paper experiment, a 1.3 billion parameter ALiBi model trained at length 1024 and evaluated at 2048 matched the perplexity of a sinusoidal model trained at 2048, while using about 11% less training time and memory in that setup.^{[7]Reference 7Train Short, Test Long: Attention with Linear Biases Enables Input Length Generalization.https://arxiv.org/abs/2108.12409} That result comes with three mechanism-level properties worth separating:

• The bias is monotonic: a farther visible key always gets a more negative penalty.
• The rule is defined for any distance, so there's no learned table to run out of.
• Fixed slopes give a spectrum from local heads to longer-range heads.

The finite entries in that matrix are a soft recency bias. Very negative logits can underflow to zero in finite precision after the softmax max-shift, but mathematically a distant visible key keeps a nonzero weight. Only the causal mask hard-zeros the future.

The final snippet tests that distinction with equal raw scores. A farther visible key should lose weight, not disappear.

Two past tokens have equal raw attention scores, but one is four positions farther from the query under ALiBi. Does ALiBi mask the farther token?

Answer

No. ALiBi subtracts a larger linear bias from the farther token before softmax, so its weight usually falls but stays nonzero. Only the separate causal mask removes future positions outright.

A closed-form bias can be evaluated at offsets the model never trained on. Whether the checkpoint still retrieves buried evidence is a different question, and long-context evaluation makes that question concrete.

---

Long context: a defined formula still needs evidence

Here’s the operational handoff: a checkpoint trained at 4K now has a 32K serving target, and a passkey may sit near token 30,000. You can write down $PE_{32000}$ or $R_{32000}$, but that only proves the position rule is defined. It doesn’t prove the model can route attention to the passkey.

Use the table as an experiment map. Each row separates what arithmetic does past the training length from what a paper actually measured.

Use two evidence gates for an extension: short-context behavior must survive, then retrieval and target tasks must work at the new length. A clean perplexity curve clears neither gate by itself.

These are different experiments, not a universal ranking. Treat context length as a model-and-eval claim, not a config value.^{[7]Reference 7Train Short, Test Long: Attention with Linear Biases Enables Input Length Generalization.https://arxiv.org/abs/2108.12409[9]Reference 9Extending Context Window of Large Language Models via Positional Interpolation.https://arxiv.org/abs/2306.15595[10]Reference 10YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071}

PI: compress coordinates into the trained range

Position Interpolation (PI) scales position indices down so they land inside the trained range:^{[9]Reference 9Extending Context Window of Large Language Models via Positional Interpolation.https://arxiv.org/abs/2306.15595}

$f'(x, m) = f\!\left(x,\ m \cdot \frac{L}{L'}\right)$

Chen et al. started from LLaMA checkpoints trained at 2048 and, with up to 1,000 fine-tuning steps, reported windows up to 32,768. The frequencies stay put; only the index that multiplies $\theta$ is compressed. In a scale-4 toy (4096 → 16384), the last target position, 16383, maps to 4095.75. Predict that endpoint before running the code.

NTK-aware scaling: stretch frequencies unevenly

PI compresses every index equally. NTK-aware scaling instead changes the frequency base, stretching low-frequency bands more than high-frequency ones:^{[10]Reference 10YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071}

$\theta'_{\text{base}} = \theta_{\text{base}} \cdot s^{d/(d-2)}$

Here $s = L_{\text{target}} / L_{\text{train}}$ and $d$ is the per-head rotary dimension. The first pair keeps $\theta_0 = 1$ for any base; later pairs slow down. Peng et al. document this as earlier NTK-aware work while developing YaRN. Quality still depends on model, target length, adaptation data, and evaluation. Code Llama uses the same base-scaling idea at an extreme, setting its base to $10^6$.^{[11]Reference 11Code Llama: Open Foundation Models for Codehttps://arxiv.org/abs/2308.12950}

YaRN: treat frequency bands differently

YaRN (Yet another RoPE extensioN) combines NTK-by-parts interpolation with an attention temperature.^{[10]Reference 10YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071} Its bands make a deliberate trade: high-frequency bands stay closer to their original wavelengths for local detail, low-frequency bands interpolate for global span, and middle bands ramp between the two. Peng et al. report Llama-2 variants evaluated through 128K, including passkey retrieval inside that window for their 7B and 13B YaRN runs.

The paper’s recommended temperature fit on its Llama experiments is $\sqrt{1/t} = 0.1 \ln(s) + 1$. They implement it by scaling the complex RoPE embeddings by that factor, equivalent to scaling $q$ and $k$ without touching the attention kernel. Treat $t$ as a hyperparameter to validate on the deployed model.

The figure compares the three choices on the same 4K-to-16K target. Look for what stays local, what stretches, and where the attention factor enters:

Llama 3 takes a related production path without PI: it raises the RoPE base to 500,000 during pretraining to support longer contexts.^{[6]Reference 6The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} That’s trained geometry, not a config flag to copy onto an arbitrary checkpoint.

---

Choose by mechanism, then earn the window

Choose the insertion point that matches the behavior you want. Then test short-context quality, long-range retrieval, generation, and target tasks at the length you plan to advertise.

The implementation choice decides where position enters attention. The checkpoint still has to earn the window you advertise.

Check the three insertion points

• Explain why unmasked self-attention without a position signal is permutation equivariant.
• Point to where sinusoids, RoPE, and ALiBi insert position: embeddings, Q/K geometry, or logits.
• Compute a RoPE relative phase and an ALiBi distance penalty on a tiny example.
• Treat configured context length as a claim that needs retrieval and task evidence at the deployed length.

Common misconceptions

• Order seems implicit: Symptom: you say unmasked attention already knows sequence order. Cause: you skipped the permutation-equivariance check. Fix: start from content-only attention, then account separately for position signals and masking.
• All methods sound the same: Symptom: sinusoids, learned tables, RoPE, and ALiBi blur together. Cause: you aren't tracking the insertion point. Fix: embeddings vs Q/K geometry vs logit bias.
• RoPE rotates the wrong tensor: Symptom: you apply RoPE to embeddings or V. Cause: routing and aggregation got merged. Fix: rotate Q and K after projection; leave V without a direct RoPE rotation.
• Long context looks solved on paper: Symptom: perplexity improves but retrieval still fails. Cause: you treated mathematical validity as behavioral proof. Fix: run passkey, retrieval, summarization, and target-task evals at the deployed length.
• Any base change is enough: Symptom: the config advertises a larger window but distant facts disappear. Cause: a new $\theta$ doesn't teach the model to use long-range evidence. Fix: pair geometry changes with an adaptation method, long-context data, and task-level checks.
• Concatenate by default: Symptom: width and projection cost balloon. Cause: you tried to keep content and position in separate channels at all costs. Fix: add position vectors unless you have a reason and a budget for a wider stack.