Positional Encoding: RoPE & ALiBi

Vision Transformers showed why image patches need position. Text has the same problem: "The package left the warehouse" and "The warehouse left the package" use nearly the same words, but the order changes the meaning.

Self-attention has a surprising blind spot: by itself, it has no built-in notion of "first", "second", or "third" token. If you permute the input sequence, the outputs permute in the same way. This is called permutation equivariance, and it means the model needs a separate signal for where each token sits.

Positional encoding injects word-order information into an architecture that would otherwise be order-blind. In this article, you'll learn why position information is needed, how the original sinusoidal approach works, and how relative methods like RoPE and ALiBi change long-context behavior.

---

The permutation equivariance problem

Self-attention computes:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

What this equation says

Compare every query with all keys ($QK^T$), scale by $\sqrt{d_k}$ (key vector dimension) for stable softmax, convert scores to attention weights, then mix value vectors $V$ using those weights.

This unmasked operation is permutation equivariant: if you shuffle the input token order, the output is shuffled in the same way. The attention weights $QK^T$ depend on content, not position.^[1]

Without positional information, transformers can't distinguish between different word orders, which can completely alter the meaning of a sentence. For example, these two sequences contain identical tokens but have very different semantics. When processed by a position-blind transformer, they produce the same attention pattern up to the same row/column permutation:

Meaning depends on word order, so the model needs a mechanism to inject position information. A causal mask also adds directional structure in decoder-only models, but it doesn't replace positional encoding: the mask tells a token which earlier slots are visible, not how far away those slots are.

This tiny attention implementation verifies the property directly. It permutes input rows, recomputes attention, and gets exactly the permuted original output.

Three families of solutions

Here is the map of approaches we will explore, so you know where each method fits before we walk through it step by step.

The methods enter the transformer at different points. The three paths below are alternatives, not steps that every model applies together.

---

Method 1: Sinusoidal positional encoding

The original approach^[1]

Think of positional encoding like reading a clock. A clock uses multiple hands moving at different speeds (seconds, minutes, hours) to uniquely identify any moment in a 12-hour period. Sinusoidal positional encoding does the same thing for words in a sequence: each position gets a unique "fingerprint" made of sine and cosine waves at different frequencies. Low dimensions oscillate quickly (like seconds), while high dimensions oscillate slowly (like hours).

For position $\text{pos}$ and dimension $i$ (where $d_{\text{model}}$ is the embedding dimension):

$PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

$PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

What the sine and cosine waves mean

The combination of these sine and cosine waves creates a unique pattern for every position. Each pair of dimensions acts like one clock hand spinning at a particular speed. Position 0 starts every hand at 0 or 1. As you move down the sequence, each hand turns a little more, and the exact angle of every hand together forms a fingerprint that no other position shares.

Why sinusoids?

Because $PE_{\text{pos}+k}$ can be expressed as a linear function of $PE_{\text{pos}}$:

$PE_{\text{pos}+k} = R_k \cdot PE_{\text{pos}}$

where $R_k$ is a rotation matrix that depends only on offset $k$, not absolute position. This means the model can, in principle, learn to attend based on relative position.^[1]

Here's how we can generate these multi-frequency sinusoidal positional encodings with plain Python. The function takes the maximum sequence length and embedding dimension as inputs to compute the required frequencies. It returns a table of positional encodings that can be added directly to token embeddings before they enter attention layers.

For the fastest pair (dimensions 0 and 1), position 2 means angle 2 radians, so we get $\sin(2) \approx 0.909$ and $\cos(2) \approx -0.416$. For the slowest pair (dimensions 6 and 7), position 2 means angle 0.002 radians, so the values stay extremely close to 0 and 1. This is the multi-scale fingerprint in action.

Frequency interpretation

The multi-scale frequencies mean the model can attend to both local and distant positions through different embedding dimensions, similar to how Fourier series decompose signals at multiple scales.

Learned vs sinusoidal absolute encodings

Instead of fixed sinusoids, models like BERT (Bidirectional Encoder Representations from Transformers) and GPT-2 (Generative Pre-trained Transformer 2) use learned position embeddings: a trainable matrix $P \in \mathbb{R}^{L_{\max} \times d}$ where each position gets its own optimizable vector.^[2][3]

Vaswani et al. (2017) tested both and found "nearly identical results."^[1] Later architectures also explore relative-position methods, which insert position into attention in different ways.

Why add instead of concatenate?

A common intuition is that concatenating the positional vector to the token embedding (creating a $2d$ vector) would perfectly preserve both pieces of information. However, concatenation would either force the whole transformer to run at width $2d$ or require an extra projection layer to get back to $d$. If you keep the model width at $2d$, square matrices like $W_Q$, $W_K$, and $W_V$ become roughly 4x larger. Addition keeps the hidden width unchanged and lets downstream learned projections use the combined representation.

Addition doesn't guarantee that content and position stay neatly separated. It keeps the hidden width fixed and gives later learned projections one combined representation to use. Whether that works well is empirical: Vaswani et al. report nearly identical results for their sinusoidal and learned-position variants.^[1]

Limitations of absolute methods

---

Method 2: Rotary Position Embeddings (RoPE)

A relative-position approach

RoPE (Rotary Position Embeddings) was introduced in RoFormer and is used in Llama-family models described in Meta's Llama 3 report.^[4][5]

Core insight

Instead of adding position to embeddings, rotate the query and key vectors in 2D subspaces. The dot product between rotated vectors naturally encodes relative position.

Imagine two dancers (a query and a key) on a circular stage. RoPE assigns each dancer a rotation angle based on their position in the sequence. When they face each other to "interact" (dot product), only the difference in their angles matters, not their absolute positions. This is why RoPE naturally encodes relative position: a dancer at position 5 interacting with position 3 looks the same as position 105 interacting with position 103.

The math

For a 2D subspace $(q_{2i}, q_{2i+1})$ at position $m$, apply rotation:

$$\begin{pmatrix} q_{2i}^{(m)} \\ q_{2i+1}^{(m)} \end{pmatrix} = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix} \begin{pmatrix} q_{2i} \\ q_{2i+1} \end{pmatrix}$$

How to read the rotation matrix

Take each consecutive pair of numbers in the query vector and spin them like a clock hand. How far they spin depends on two things: the token's position $m$ (further in the sequence = more rotation) and the frequency $\theta_i$ (each pair rotates at a different speed). The result: every position creates a unique rotational pattern that the model can use to figure out "how far apart are these two words?"

The frequency base follows the same pattern as sinusoidal encoding, but production RoPE usually applies it over the per-head rotary dimension, not the full model width:

$\theta_i = 10000^{-2i/d_{\text{rotary}}}$

If every head dimension is rotated, $d_{\text{rotary}} = d_k$. Some architectures rotate only part of each head, so the config's rotary dimension is the value that matters.

The key property

When computing $q^{(m)} \cdot k^{(n)}$, the rotation matrices combine:

$q^{(m)T} k^{(n)} = q^T R_{\theta}^{(n-m)} k$

Toy numerical example (2D subspace). Suppose a query vector pair at position 5 with frequency $\theta = 1.0$ for simplicity, and a key at position 2:

Running this shows that the interaction depends only on the offset, exactly as the math predicts. This relative-position property is useful, but alone provides no proof that a model works beyond its trained context window.

Why this matters

The attention score between a query at position $m$ and a key at position $n$ depends only on the signed relative offset $(n - m)$ between them, not on their absolute positions. Token 5 attending to token 3 produces the same positional structure as token 1005 attending to token 1003. This is what makes RoPE "relative": it captures offset, not absolute address.

Worked example: the same distance always gives the same score

Picture a single 2D pair where both the query and the key start as the simple unit vector $[1, 0]$. Let the rotation frequency for this pair be $\theta = 0.5$ radians.

• Query at position $m = 2$, key at position $n = 1$: relative distance $d = n - m = -1$. After rotation, the dot product equals $\cos(d \cdot \theta) = \cos(-0.5) \approx 0.878$.

• Query at position $m = 102$, key at position $n = 101$: relative distance is still $d = -1$. After rotation, the dot product is again $\cos(-0.5) \approx 0.878$.

The absolute positions changed by 100, but the positional part of the attention score stayed identical because only the distance between the tokens matters. If the distance were $d = -2$, this particular pair would score $\cos(-1.0) \approx 0.540$, which is smaller. Avoid generalizing that single comparison into a monotonic distance penalty: cosine phases can rise again at larger offsets, and a real head sums many frequency pairs.

Implementation

The code below demonstrates how RoPE applies these rotations in practice. It takes the query and key tensors as inputs, along with precomputed rotation frequencies based on the sequence length. The function reshapes the vectors into 2D pairs, applies the complex rotations, and returns the rotated query and key tensors.

What RoPE changes

RoPE directly rotates Q and K, not V. This places its relative-position operation in attention routing rather than applying the same rotation to aggregated values. In deeper layers, V still comes from hidden states that already include context mixed by earlier position-aware attention.

Relative offset isn't a monotonic penalty

RoPE gives attention a structured function of relative offset. Farther tokens aren't guaranteed to receive smaller positional contributions. For a single pair of identical unit vectors, the positional score is:

$S_{\Delta} = \cos(\Delta\theta)$

This quantity oscillates as distance $\Delta$ changes. A full head combines multiple frequencies and content-dependent coefficients, so its score can also rise or fall across offsets.

The RoFormer paper analyzes a long-term decay bound aggregated over RoPE frequencies, and Men et al. (2024) study how the RoPE base affects long-context retrieval behavior.^[4][7] Those analyses motivate careful base selection and evaluation. They remain distinct from ALiBi's explicit linear distance penalty.

Important caution: simply increasing the base frequency can give "superficial" long-context capability in the evaluations reported by Men et al.: perplexity may look good while retrieval accuracy degrades.^[7] A changed RoPE configuration therefore needs retrieval and task evaluation at its deployed length.

---

Method 3: Attention with Linear Biases (ALiBi)

The simplest approach^[6]

ALiBi (Attention with Linear Biases) doesn't modify embeddings at all. Instead, it adds a head-specific linear bias directly to attention scores:

Imagine you're at a crowded party. People standing right next to you speak at full volume, but the farther away someone stands, the quieter they sound, linearly. ALiBi works like this: it applies a simple distance penalty to attention scores, so nearby tokens "speak loudly" and distant tokens are naturally muffled. Different attention heads act like listeners with different hearing ranges: some focus on nearby whispers, others can hear across the room.

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + B^{(h)}\right) V,\quad B^{(h)}_{ij} = -m_h(i-j)\ \text{for}\ j \le i$

What the ALiBi formula does

This is the normal attention formula with one modification: before applying softmax, we add a negative bias that gets more negative the farther back a key token sits from the current query. In a causal decoder, the current token gets zero penalty, one token back gets $-m_h$, two tokens back gets $-2m_h$, and so on. A steep slope makes the head short-sighted. A gentle slope lets it see further.

Here, $m_h$ is the slope for head $h$. Future positions still get removed by the causal mask.

Head slopes

For $n$ heads, the paper uses a fixed geometric schedule. For power-of-two head counts, this reduces to a clean sequence like $m_k = 2^{-8k/n}$.

For 8 heads (n=8): $m = \frac{1}{2}, \frac{1}{4}, \frac{1}{8}, \frac{1}{16}, \frac{1}{32}, \frac{1}{64}, \frac{1}{128}, \frac{1}{256}$.

Different heads get different slopes through a fixed schedule, giving the model a spectrum of attention ranges, from very local (steep slope) to nearly global (gentle slope). For non-power-of-two head counts, the reference implementation uses a recursive workaround so slopes remain well spread.^[6]

Worked example: a concrete distance penalty

Picture the token "left" at position 3 attending back to earlier words in "The package left the warehouse." The raw dot-product score for "package" at position 1 is 2.5. Head 4 has slope $m = 1/16 \approx 0.0625$.

• Distance from "left" to "package" = $3 - 1 = 2$
• Penalty = $-0.0625 \times 2 = -0.125$
• Score entering softmax = $2.5 - 0.125 = 2.375$

For "The" at position 0, the distance is 3, so the penalty is $-0.0625 \times 3 = -0.1875$, and the score entering softmax is $2.3125$. The farther token is slightly quieter, but not silenced. With Head 1 ($m = 0.5$), the same distances would produce penalties of $-1.0$ and $-1.5$, making the head far more myopic.

The following snippet shows how to construct the ALiBi bias tensor for a causal decoder. It follows the reference slope helper, including non-power-of-two head counts, and bakes the causal mask into the returned bias matrix.

What ALiBi demonstrated

In one original-paper experiment, a 1.3 billion parameter ALiBi model is trained at length 1024 and evaluated at length 2048, matching the perplexity of a sinusoidal model trained at length 2048 while using less training time and memory in that setup.^[6] The mechanism has several useful properties:

• Linear biases are monotonic: distant tokens always get penalized more
• The bias rule is defined for any distance, so there's no learned position table to run out
• No learned position parameters -> nothing needs to be interpolated
• The fixed slopes create a natural spectrum from local heads to longer-range heads

Softmax numerical stability with ALiBi

ALiBi interacts elegantly with standard softmax optimizations. ALiBi biases can produce very negative values for distant tokens. In practice, softmax is computed using the max-shift trick to prevent numerical overflow:

$\text{softmax}(s_i + b_i) = \frac{e^{s_i + b_i - \max(s + b)}}{\sum_j e^{s_j + b_j - \max(s + b)}}$

Where $s_i$ is the raw attention score and $b_i$ is the ALiBi distance bias for token $i$.

When computing this in finite precision, very negative ALiBi biases can make distant tokens' shifted exponent values extremely small. Conceptually ALiBi is still a soft distance-dependent recency bias, not a hard window. The causal mask remains separate, and every distant past weight remains mathematically possible.

---

The extrapolation challenge

This is a critical practical challenge in production systems:

"Your model was trained on 4K tokens. How does it perform at 32K?"

These are results from different experiments, not a universal ranking. Context length is a model-and-evaluation claim, not a property proved by selecting a positional formula.^[6][8][9]

RoPE context extension techniques

When adapting a RoPE-based model for longer contexts, these methods illustrate different interventions:

1. Position Interpolation (PI)

Scale position indices down to fit within training range^[8]

$\text{Let } L = \text{train length},\ L' = \text{target length}$ $f'(x, m) = f(x, m \cdot L / L')$

What rescaling the position means

If you trained on 4K tokens but want to handle 16K, you shrink all position indices by 4x so they fit within the range the model already knows. Position 16,000 gets mapped to effective position 4,000. Critically, the rotation frequencies theta themselves stay unchanged. Only the position index that multiplies theta gets rescaled. Chen et al. report extensions to 32K with up to 1,000 fine-tuning steps in their Llama experiments.^[8]

2. NTK (Neural Tangent Kernel)-aware Scaling

Scale the frequency base instead of positions, trading off between low and high frequency components^[9]

$\theta'_{\text{base}} = \theta_{\text{base}} \cdot s^{d/(d-2)}$

where $s = L_{\text{target}} / L_{\text{train}}$ is the scale factor and $d$ is the per-head RoPE dimension (the head dimension, not the full hidden size).

What scaling the base changes

Instead of compressing every position, this formulation changes the base so lower-frequency bands are stretched more than high-frequency bands. Peng et al. document this as earlier NTK-aware work while developing YaRN; any quality claim remains dependent on model, target length, adaptation data, and evaluation.^[9]

3. YaRN (Yet another RoPE extensioN)

Combines NTK-by-parts interpolation with attention temperature adjustment.^[9] Peng et al. report RoPE-based Llama-family variants evaluated at contexts up to 128K.

The paper's recommended temperature rule for its Llama experiments is $\sqrt{1/t} = 0.1 \ln(s) + 1$, where $s = L_{\text{target}} / L_{\text{train}}$. NTK-by-parts interpolates frequency bands differently; temperature adjustment changes attention scale. Both choices remain hyperparameters to validate on the deployed model.

---

Choosing a positional method

The implementation choice decides where position enters attention. It still leaves the trained checkpoint to be evaluated on the exact lengths and tasks being served.

---

Mastery check

Key concepts

Evaluation rubric

• Strong: can derive why RoPE makes the attention score depend on relative offset, can explain why V stays unrotated, and can pick a context-extension method plus validation plan for a real deployment.
• Good: can compare sinusoidal, learned, RoPE, and ALiBi correctly, but still needs help connecting the math to serving tradeoffs.
• Needs work: still mixes up where position enters the model, or treats a longer configured context window as proof that long-context retrieval will work.

Follow-up questions

---

What to remember

Without position encodings, swapping inputs just swaps outputs. Sinusoidal encodings add multi-frequency position fingerprints to token embeddings. RoPE rotates Q and K so the attention dot product can represent relative offset without directly rotating V. ALiBi adds distance penalties directly to attention logits. Long-context claims need validation at the target length, because a method can be mathematically defined for a longer sequence and still fail retrieval behavior.

Common pitfalls

• Order still looks like magic: Symptom: you say attention already knows sequence order. Cause: you skipped the permutation-equivariance step. Fix: start from the fact that self-attention is content-based unless a position signal is injected.
• All methods sound same: Symptom: sinusoidal, learned, RoPE, and ALiBi blur into one idea. Cause: you aren't tracking where position enters. Fix: separate embeddings, Q/K geometry, and attention-logit bias explicitly.
• RoPE rotates wrong tensor: Symptom: you describe applying RoPE rotation to embeddings or V. Cause: you lost the routing-versus-aggregation distinction. Fix: rotate Q and K after projection; leave V without direct RoPE rotation.
• Long context looks solved on paper: Symptom: perplexity improves but retrieval still fails. Cause: you treated mathematical validity as behavioral proof. Fix: run passkey, retrieval, summarization, and target-task evals at deployed length.
• Bigger base must be enough: Symptom: config advertises a larger window but distant facts still disappear. Cause: base change alone doesn't teach the model to use long-range evidence. Fix: pair geometry changes with rescaling method, long-context data, and task-level checks.
• Concatenate by default: Symptom: width and projection cost balloon without a clear gain. Cause: you tried to preserve content and position separately at all costs. Fix: add position vectors unless you have a strong reason and budget for a wider architecture.

Going deeper

Why not concatenate positional encodings?

Concatenation would either force the transformer to run at width $2d$ or require an extra projection layer to get back to $d$. If you keep the model width at $2d$, square projection matrices like $W_Q$, $W_K$, and $W_V$ become roughly 4x larger. Addition keeps the width constant while learned projections operate on the resulting combined representation.

RoPE relative derivation

At position $m$, $q^{(m)} = R(m\theta) \cdot q$. At position $n$, $k^{(n)} = R(n\theta) \cdot k$. $q^{(m)T} k^{(n)} = q^T R(m\theta)^T R(n\theta) k = q^T R((n-m)\theta) k$

After rotation, the dot product depends only on the relative offset $(n-m)$, not on absolute positions. This is why RoPE naturally encodes relative position information.

Why is RoPE base size part of a long-context design?

Men et al. derive and evaluate a relationship between RoPE base and effective long-context retrieval, while also showing that perplexity alone can hide retrieval failures.^[7] Base size is therefore a parameter to choose and validate alongside adaptation data and task-level long-context evaluations, not a standalone certificate of usable context length.