4. Position
Draft
Attention reads a set
The dog bit the man and the man bit the dog are made of the same five words. To a head of attention as Section 3 defined it they are the same input: the head compares every query with every key and averages values by the weights that result, and nothing in that computation knows which token came first.
as written The dog bit the man . reordered The man bit the dog . one attention head, made-up weights: how far each word's output moved from the output the same word had in the sentence as written The man bit the dog . no mask, no positions 0 0 0 0 0 0 0 of 6 changed causal mask 0 0.30 0.34 0.63 0.16 0 4 of 6 mask, rotary positions 0 0.15 0.27 0.15 0.74 0.61 5 of 6 without mask or positions the head reads a set of words; under the mask a word reads the set to its left; with positions, also how far away each is (the live figure needs JavaScript; it loads no data)
With no causal mask and no position signal, every word leaves the head with the output it had in the sentence as written: exchange two words and their outputs are exchanged, and nothing else moves. Under the causal mask outputs change, because a word now reads only the words to its left, and the two orders put different words there. The full stop does not change: the same five words stand to its left in both sentences, and it cannot tell in which order until the head is given positions. These are the two halves of this section. Order has to be put into a transformer; and a causal language model has, through its mask, been given a little of it already.
Every position scheme since 2017 answers two questions: where position enters the computation (the input, the score, the query and key, or nowhere), and which symmetry it respects (the absolute index of a token, or the offset between two tokens). The field moved from a vector added once at the input, indexed by absolute position, to a rotation applied to queries and keys in every layer and felt only through offsets. For each scheme on the way we give the score it induces as a function of the offset, the evidence, and the failure that the next scheme answered. gpt-pp uses the rotary scheme, and the ladder of small models trains the alternatives on the same tokens (Section 4.5).
Throughout, a sequence has \(n\) tokens at positions \(1, \dots, n\). The residual stream, the running sum of vectors that every layer reads and adds to, has width \(d\), and a head has dimension \(d_h\), an even number. The vector of token \(i\) entering a layer is \(x_i \in \mathbb R^d\); in one head its query, key and value are \(q_i = W_Q x_i\), \(k_i = W_K x_i\) and \(v_i = W_V x_i\) in \(\mathbb R^{d_h}\); the score of query \(i\) against key \(j\) is \(s_{ij} = q_i^{\top} k_j/\sqrt{d_h}\), and the attention weights \(a_{ij}\) are the softmax of row \(i\) of the scores. In a causal model \(j \le i\), and \(r = i - j \ge 0\) is the offset.
Definition 4.1.1 (a head without position). For \(X \in \mathbb R^{n \times d}\) with rows \(x_1^{\top}, \dots, x_n^{\top}\), let \(S(X) = X W_Q^{\top} W_K X^{\top}/\sqrt{d_h}\) be the \(n \times n\) matrix of scores, and
\[\mathrm{Attn}(X) \;=\; \sigma\bigl(S(X)\bigr)\, X\, W_V^{\top} W_O^{\top}, \tag{4.1}\]where \(\sigma\) is the softmax of each row and \(W_O \in \mathbb R^{d \times d_h}\) writes the output of the head into the stream. The causal head uses \(S(X) + C\) in place of \(S(X)\), where the mask \(C\) has \(C_{ij} = 0\) for \(j \le i\) and \(C_{ij} = -\infty\) for \(j > i\).
Proposition 4.1.2 (permutation equivariance). For every \(n \times n\) permutation matrix \(\Pi\), \(\mathrm{Attn}(\Pi X) = \Pi\, \mathrm{Attn}(X)\). The same holds for any stack of blocks built from heads (4.1), MLPs, normalization layers and residual sums, with no mask and no position signal.
Proof. The scores transform as \(S(\Pi X) = \Pi\, S(X)\, \Pi^{\top}\): rows and columns are permuted together. Permuting the entries of a row permutes its softmax in the same way, because the normalizing sum is symmetric, so \(\sigma(\Pi S \Pi^{\top}) = \Pi\, \sigma(S)\, \Pi^{\top}\). Hence \(\mathrm{Attn}(\Pi X) = \Pi\, \sigma(S)\, \Pi^{\top} \Pi\, X W_V^{\top} W_O^{\top} = \Pi\, \mathrm{Attn}(X)\), since \(\Pi^{\top} \Pi = I\). An MLP (the feed-forward sublayer of a block, Section 5) and a normalization layer act on each row of \(X\) by itself and so commute with \(\Pi\); sums and compositions of maps that commute with \(\Pi\) commute with \(\Pi\). ∎
So a transformer with no mask and no position signal reads a multiset with one marked element, and a language model built this way would give the same next-token distribution to every reordering of the words before the last.
The causal mask breaks the symmetry, but one layer gains little from it: the output \(\sum_{j \le i} a_{ij}\, W_O v_j\) at position \(i\) is symmetric in the tokens before it, a bag of words again, the prefix. The bags are nested, though, and the next layer reads the outputs of this one, so a stack of two causal layers can tell the dog from the man. How much more a stack can do is a theorem whose first step fits in a few lines.
Proposition 4.1.3 (a causal head can count). Suppose every sequence begins with a start token (BOS, for beginning of sequence) that occurs nowhere else. There are an embedding and weights of one causal head, with no position signal, for which the head writes \(1/i\) into a chosen coordinate of the stream at every position \(i\).
Proof. Let the first coordinate of the embedding be \(1\) for the start token and \(0\) for every other token, and the second \(0\) for all. Take \(W_Q = 0\): all scores are zero and the causal softmax is uniform, \(a_{ij} = 1/i\) for \(j \le i\). Choose \(W_V\) and \(W_O\) with \(W_O W_V = e_2 e_1^{\top}\), which reads the first coordinate and writes it to the second. The output at position \(i\) is \(\sum_{j \le i} \tfrac1i\, (x_j)_1\, e_2 = \tfrac1i\, e_2\), because only \(j = 1\) carries the start token. ∎
The count \(1/i\) is a code for the absolute position. An MLP can turn it into any other code, since one hidden layer of rectified linear units interpolates any function on the finitely many points \(1, \tfrac12, \dots, \tfrac1n\); and once absolute positions are in the stream, later layers can realize scores of the form content plus a function of \(i - j\).Kazemnejad, Padhi, Natesan Ramamurthy, Das and Reddy, "The impact of positional encoding on length generalization in transformers", NeurIPS 2023 (arXiv 2305.19466), Theorems 1 and 2; the proof above follows the construction of their Appendix C.1. The idea was first stated as a conjecture, with the measurement that prompted it:
Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models, and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. We conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position. Our findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism, but also from the effects of the causal mask.
—Haviv, Ram, Press, Izsak and Levy, "Transformer language models without positional encodings still learn positional information" (2022). Findings of EMNLP 2022 (arXiv 2203.16634), abstract. The perplexities that follow are from their Table 2; at 1.3B parameters the four are 13.10, 13.05, 12.93 and 12.51.
On the corpus called The Pile, at 1,024 tokens, their model of 125M parameters with no positional encoding reaches a validation perplexity of 22.15 (the exponential of the loss per token; lower is better), against 22.04 with a learned table of position vectors, 21.49 with the sinusoids of 2017 and 19.94 with ALiBi, a relative scheme of Section 4.4. The model with no positions is last, by little, as it is again at 1.3B parameters; in our own small models it is last by more (Section 4.5). A bidirectional model without positions does much worse in the same paper: it has no mask to count with.
Remark 4.1.4 (the count needs an anchor, and a count is not a pointer). The start token does the work in Proposition 4.1.3. Feed the model one token repeated \(n\) times, with no start token: every key of the first layer is the same vector, so every head attends uniformly; every value is the same, so every position receives the same output; by induction all positions carry identical vectors to the top, and the model predicts the same thing after three repetitions as after three hundred. For the same reason no head of such a model can put its weight on the previous token of that sequence. A head that does so on any sequence, a previous-token head, is the first component of the induction circuits of Section 9, and with rotary embeddings a head can be one (Proposition 4.4.6); the heads of Example 4.5.3 bear out both halves.Barbero, Vitvitskyi, Perivolaropoulos, Pascanu and Veličković, "Round and round we go! What makes rotary positional encodings useful?", arXiv 2410.06205, 2024, Proposition 5.2 and Theorem 5.3: "NoPE is instead provably not able to construct such patterns." NoPE is no positional encoding.
Absolute positions: clocks and tables
The first transformer put order in at the bottom of the stack, once, by adding to the embedding of every token a vector that depends on its position. The section of the 2017 paper that does it is short enough to read whole.
3.5 Positional Encoding
Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add "positional encodings" to the input embeddings at the bottoms of the encoder and decoder stacks. The positional encodings have the same dimension \(d_{\text{model}}\) as the embeddings, so that the two can be summed. There are many choices of positional encodings, learned and fixed [9].
In this work, we use sine and cosine functions of different frequencies:
\[PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})\]
\[PE_{(pos,2i+1)}=cos(pos/10000^{2i/d_{\text{model}}})\]
where \(pos\) is the position and \(i\) is the dimension. That is, each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from \(2\pi\) to \(10000\cdot 2\pi\). We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset \(k\), \(PE_{pos+k}\) can be represented as a linear function of \(PE_{pos}\).
We also experimented with using learned positional embeddings [9] instead, and found that the two versions produced nearly identical results (see Table 3 row (E)). We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training.
Definition 4.2.1 (sinusoidal encoding). For even \(d\) let \(\omega_t = 10000^{-2t/d}\) for \(t = 0, \dots, d/2 - 1\). The encoding of position \(i\) is the vector \(p_i \in \mathbb R^d\) with
\[(p_i)_{2t} = \sin(\omega_t\, i), \qquad (p_i)_{2t+1} = \cos(\omega_t\, i), \tag{4.2}\]and the first block reads \(x_i = w_i + p_i\), where \(w_i\) is the embedding of the token at position \(i\). (The \(pos\) and \(i\) of the paper are our \(i\) and \(t\).)
In the code. The table of (4.2), one row for each position. It is sinusoidal_table in gpt-pp's model.py.
def sinusoidal_table(n_positions: int, d: int) -> Tensor:
...
position = torch.arange(n_positions, dtype=torch.float64).unsqueeze(1)
frequency = 10000.0 ** (-torch.arange(0, d, 2, dtype=torch.float64) / d)
table = torch.zeros(n_positions, d, dtype=torch.float64)
table[:, 0::2] = torch.sin(position * frequency)
table[:, 1::2] = torch.cos(position * frequency)
return table.float()
frequency holds the rates \(\omega_t\), and row \(i\) of table is the vector \(p_i\), with the sines in the even columns and the cosines in the odd ones. The first row is position \(0\), where this section counts positions from \(1\). The arithmetic is done in float64 and the table is returned in float32. The sum \(x_i = w_i + p_i\) is not in this function. In the model it has a scale and a place that the definition does not give: the table is multiplied by the standard deviation at which the token embedding is drawn, and the recipe of gpt-pp normalizes each token vector after the embedding, so that it matters whether the table is added before that normalization or after it. Section 4.5 shows both.
Each pair of coordinates is the hand of a clock: a unit vector that turns by \(\omega_t\) radians for every token. The rates are geometric: the fastest clock turns one radian per token, a turn every \(2\pi \approx 6.28\) tokens, and the slowest takes about \(2\pi \cdot 10^4\) tokens. At the width of the ladder models of this series, \(d = 512\), there are 256 clocks with periods from 6.28 to 60,611 tokens, each 3.7% longer than the one before, and 123 of them complete a turn inside the ladder's context of 512 tokens. A position is read off them as a time is read off second, minute and hour hands: the fast hands separate neighbors and soon repeat, the slow hands separate distant positions.
the 2017 encoding at 32 dimensions as 16 clocks: a pair (sin, cos) is the tip of a hand position p = 300, position p + k = 305, offset k = 5 clock 1 2 3 4 5 6 7 8 ... 16 tokens a turn 6.3 11 20 35 63 112 199 353 ... 35,333 turn by k = 5 286° 161° 91° 51° 29° 16° 9.1° 5.1° ... 0.1° cos of the turn 0.28 −0.95 −0.01 0.63 0.88 0.96 0.99 1.00 ... 1.00 sum of the 16 cosines = PE(p) · PE(p + k) = 11.78 at every p (16 at k = 0) p = 300 as a binary counter, lowest bit first (bit b cycles every 2^b tokens) 0 0 1 1 0 1 0 0 1 0 0 (the live figure needs JavaScript; it loads no data)
At 32 dimensions the encoding has 16 clocks. Five tokens on, the first hand has turned by 286°, the second by 161°, the third by 91°, wherever the first position is. The dot product of two encodings, the sum of the cosines of those turns, depends on the distance alone.
Proposition 4.2.2 (a shift is a rotation). For every real \(k\) there is an orthogonal matrix \(T_k\), block-diagonal on the pairs of (4.2) and independent of \(i\), with \(p_{i+k} = T_k\, p_i\) for all \(i\); and \(T_k T_l = T_{k+l}\).
Proof. By the addition formulas,
\[\begin{pmatrix}\sin\omega_t(i+k)\\ \cos\omega_t(i+k)\end{pmatrix} = \begin{pmatrix}\cos\omega_t k & \sin\omega_t k\\ -\sin\omega_t k & \cos\omega_t k\end{pmatrix}\begin{pmatrix}\sin\omega_t i\\ \cos\omega_t i\end{pmatrix},\]a rotation of the pair \(t\) by the angle \(\omega_t k\); rotations of a plane compose by adding their angles. ∎
Proposition 4.2.3 (the kernel of the encoding). \(\langle p_i, p_j\rangle = K(i - j)\) with \(K(r) = \sum_t \cos(\omega_t r)\) and \(K(0) = d/2\). For large \(d\) and \(1 \ll r \ll 10^4\),
\[\frac{K(r)}{K(0)} \;\approx\; 1 - \frac{\ln r + \gamma}{\ln 10^4}, \tag{4.3}\]where \(\gamma \approx 0.577\) is Euler's constant: the kernel loses a quarter of its height for every factor of ten in the distance.
Proof. In each pair \(\sin a \sin b + \cos a \cos b = \cos(a - b)\). For (4.3), the sum is \(d/2\) times a Riemann sum of \(\int_0^1 \cos(r\, 10^{-4u})\, du\), and the substitution \(w = r\, 10^{-4u}\) turns the integral into a difference of two cosine integrals, \((\mathrm{Ci}(r) - \mathrm{Ci}(r/10^4))/\ln 10^4\), with \(\mathrm{Ci}(x) \approx 0\) for large \(x\) and \(\mathrm{Ci}(x) \approx \gamma + \ln x\) for small \(x\). ∎
With 16 clocks the sum ripples; for the 256 clocks of a width of 512, (4.3) describes it to within 0.07 of the exact ratio at every distance from 3 to 3,000 tokens. The similarity of two positions falls as a straight line in the logarithm of their distance: a scale-free ruler, on which the step from 10 to 20 tokens counts as much as the step from 1,000 to 2,000. A later model, T5, will cut the same ruler into buckets by hand, and rotary embeddings contain it.
What does a head make of these vectors? Up to the factor \(1/\sqrt{d_h}\), the score of the first layer between positions \(i\) and \(j\) is
\[(w_i + p_i)^{\top} M\, (w_j + p_j) \;=\; \underbrace{w_i^{\top} M w_j}_{\text{content, content}} + \underbrace{w_i^{\top} M p_j}_{\text{content, position}} + \underbrace{p_i^{\top} M w_j}_{\text{position, content}} + \underbrace{p_i^{\top} M p_j}_{\text{position, position}}, \qquad M = W_Q^{\top} W_K . \tag{4.4}\]Only the last term can prefer an offset by itself, and one matrix \(M\) serves all four. The hypothesis was that Proposition 4.2.2 would make an offset easy to learn. The next statement says how much has to be learned.
Proposition 4.2.4 (when the sinusoid gives a relative score). The term \(p_i^{\top} M p_j\) is a function of \(i - j\) for all integers \(i, j\) if and only if \(M\) commutes with the shift \(T_1\), that is, if and only if \(M\) is block-diagonal on the pairs of (4.2) with every block a scaled rotation \(\begin{pmatrix}\alpha_t & \beta_t\\ -\beta_t & \alpha_t\end{pmatrix}\). Then \(p_i^{\top} M p_j = \sum_t \bigl(\alpha_t \cos\omega_t(i - j) + \beta_t \sin\omega_t(i - j)\bigr)\).
Proof. By the product formulas, the block of \(M\) between the pairs \(t\) and \(t^{\prime}\) contributes cosines and sines of \(\omega_t i \pm \omega_{t^{\prime}} j\). The rates are distinct and lie in \((0, 1]\), so these functions of \((i, j)\) are linearly independent, and only those of \(\omega_t (i - j)\) are unchanged when \(i\) and \(j\) advance together. The other coefficients must vanish, which leaves diagonal blocks with equal diagonal and opposite off-diagonal entries: the matrices that commute with \(T_1\). ∎
These matrices form a subspace of dimension \(d\) among the \(d^2\) entries of \(M\): 512 of 262,144 at \(d = 512\). Gradient descent may come near it, but nothing in the architecture prefers it, and the two cross terms of (4.4) depend on absolute positions whatever \(M\) is. The hypothesis was a hope; rotary embeddings turn it into a constraint (Section 4.4).
The alternative that the passage mentions, its reference [9], is a month older than the transformer: a convolutional translation model that learned one free vector for each position and added it to the word embedding, to give the model, in the words of its authors, "a sense of order".Gehring, Auli, Grangier, Yarats and Dauphin, "Convolutional sequence to sequence learning", arXiv 1705.03122, 2017, Sections 3.1 and 5.4; removing the tables cost 0.5 BLEU, a score of translation quality, and the tables "impose a restriction on the maximum sentence length".
Definition 4.2.5 (learned absolute embedding). A table \(P \in \mathbb R^{L_{\max} \times d}\) of trained parameters; the first block reads \(x_i = w_i + P_i\). There is no row for \(i > L_{\max}\).
GPT made this choice in 2018 in one sentence, "We used learned position embeddings instead of the sinusoidal version proposed in the original work";Radford, Narasimhan, Salimans and Sutskever, "Improving language understanding by generative pre-training", OpenAI, 2018, Section 4.1, page 5. the input of the bidirectional encoder BERT, the same year, is the sum of a token, a segment and a position embedding, and the released GPT-2 carries a table of 1,024 positions.Devlin, Chang, Lee and Toutanova, "BERT: pre-training of deep bidirectional transformers for language understanding", NAACL 2019 (arXiv 1810.04805), caption of Figure 2. For GPT-2, the configuration released with the model. The models of this series that follow the 2019 recipe, the design of GPT-2, do the same, with 786,432 parameters at the size of gpt-pp, 262,144 on the ladder and 32,768 in the smallest model, under 2% of each. The evidence for either choice was thin: in the ablation of the 2017 paper the two tie on perplexity, 4.92 each, and differ by 0.1 BLEU (a score of translation quality; higher is better), and in the language models of Section 4.1 the sinusoid is a little ahead, as it is in ours (Section 4.5).Vaswani et al. (2017), Table 3, the base model and row (E): 25.8 for the sinusoid and 25.7 for the table, on the English-to-German development set.
The additive schemes left three things open. The first is length: a learned table has no row for position \(L_{\max} + 1\), and the sinusoid has a vector there that the model has never seen. The hope of extrapolation did not survive measurement: on the WikiText-103 benchmark a sinusoidal model trained on 512 tokens keeps improving for about 20 further tokens and then degrades.Press, Smith and Lewis (2021), Sections 2.2 and 4.1 and Figure 1; cited in full in Section 4.4. The second is the mixing of position into content: the sum puts what and where into one vector, and (4.4) passes both through one matrix. Ke, He and Liu looked at the four terms in a trained BERT:
In this work, we investigate the positional encoding methods used in language pre-training (e.g., BERT) and identify several problems in the existing formulations. First, we show that in the absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations between the two heterogeneous information resources. It may bring unnecessary randomness in the attention and further limit the expressiveness of the model. Second, we question whether treating the position of the symbol [CLS] the same as other words is a reasonable design, considering its special role (the representation of the entire sentence) in the downstream tasks. Motivated from above analysis, we propose a new positional encoding method called Transformer with Untied Positional Encoding (TUPE). In the self-attention module, TUPE computes the word contextual correlation and positional correlation separately with different parameterizations and then adds them together. This design removes the mixed and noisy correlations over heterogeneous embeddings and offers more expressiveness by using different projection matrices. Furthermore, TUPE unties the [CLS] symbol from other positions, making it easier to capture information from all positions. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of the proposed method. Codes and models are released at https://github.com/guolinke/TUPE.
—Ke, He and Liu, "Rethinking positional encoding in language pre-training" (2020). ICLR 2021 (arXiv 2006.15595), abstract; in Section 3.1, of the cross terms of (4.4) in a pre-trained BERT: "We find that the second and the third term looks uniform across positions, as shown in Figure 2."
The third is the symmetry: nothing makes the score depend on \(i - j\) (Proposition 4.2.4), so a rule learned at one pair of positions is not known at another.
Relative positions
Why should a score depend on the offset between two tokens and not on their positions? Because the right answer does. If the tokens come from a stationary process, one whose statistics do not change under a shift in time, and a window is cut from it at a start independent of the text, the best prediction after position \(i\) is the conditional law of the next token given the \(i\) tokens before it, and that law is the same function of those tokens, listed by their offset from the current one, wherever the window was cut. Position enters only as the number of offsets available, the count of Proposition 4.1.3, which a causal model has for nothing.
Text is not stationary, but training windows are close to it. Every document of the corpus of gpt-pp begins with a start token, the documents are concatenated in a shuffled order, and the stream is cut into windows without regard to where documents begin. The absolute index of a token in its window then says how much context there is and nothing about the text, while an offset means the same thing everywhere. A regularity such as an adjective is followed by its noun is a statement about offset 1: a model with absolute positions has to learn it at each of the 511 adjacent pairs of positions of a window of 512 tokens, and a score that depends on the offset shares it across them.
Definition 4.3.1 (relative position). A position scheme is relative if every score is a function \(g(x_i, x_j, i - j)\) of the vectors of the two tokens and of their offset.
Three designs made scores relative before 2021, each by editing the expansion (4.4).
The first, of Shaw and others in 2018, adds to each key a learned vector chosen by the offset, clipped at a largest distance \(c\) (the \(k\) of the paper; we need \(k_j\) for the key):
\[s_{ij} = \frac{q_i^{\top}\bigl(k_j + a_{\mathrm{clip}(j - i,\, c)}\bigr)}{\sqrt{d_h}}, \qquad \mathrm{clip}(x, c) = \max(-c, \min(c, x)), \tag{4.5}\]with \(2c + 1\) vectors \(a_{-c}, \dots, a_c \in \mathbb R^{d_h}\) in each layer. The score it induces is \(q_i^{\top} a_{\mathrm{clip}(-r, c)}/\sqrt{d_h}\): a function of the offset that the query can modulate, constant beyond \(c\) tokens.
For linear sequences, edges can capture information about the relative position differences between input elements. The maximum relative position we consider is clipped to a maximum absolute value of \(k\). We hypothesized that precise relative position information is not useful beyond a certain distance. Clipping the maximum distance also enables the model to generalize to sequence lengths not seen during training. Therefore, we consider \(2k+1\) unique edge labels.
—Shaw, Uszkoreit and Vaswani, "Self-attention with relative position representations" (2018). NAACL 2018 (arXiv 1803.02155), Section 3.2; the score is their Equation 4, and the results are from their Table 1 and Sections 3.3, 4.1 and 4.3.
On the WMT 2014 translation benchmark from English to German the change was worth 0.3 BLEU for the base model (26.5 to 26.8) and 1.3 for the big one (27.9 to 29.2), for about 7% fewer training steps per second. What it left: a table in every layer, no term free of content, and vectors learned from nothing, the geometric clocks of 2017 thrown away.
The second design came from a different need. Transformer-XL reads a long text in consecutive segments of \(L\) tokens and lets each segment attend to the hidden states computed for the one before. With absolute encodings both segments are numbered from \(1\) to \(L\):
While we found the idea presented in the previous subsection very appealing, there is a crucial technical challenge we haven’t solved in order to reuse the hidden states. That is, how can we keep the positional information coherent when we reuse the states? Recall that, in the standard Transformer, the information of sequence order is provided by a set of positional encodings, denoted as \({\mathbf{U}}\in\mathbb{R}^{L_{\text{max}}\times d}\), where the \(i\)-th row \({\mathbf{U}}_{i}\) corresponds to the \(i\)-th absolute position within a segment and \(L_{\text{max}}\) prescribes the maximum possible length to be modeled. Then, the actual input to the Transformer is the element-wise addition of the word embeddings and the positional encodings. If we simply adapt this positional encoding to our recurrence mechanism, the hidden state sequence would be computed schematically by
\[{\mathbf{h}}_{\tau+1}=f({\mathbf{h}}_{\tau},{\mathbf{E}}_{ {\mathbf{s}}_{\tau+1}}+{\mathbf{U}}_{1:L})\] \[{\mathbf{h}}_{\tau}=f({\mathbf{h}}_{\tau-1},{\mathbf{E}}_{ {\mathbf{s}}_{\tau}}+{\mathbf{U}}_{1:L}),\]where \({\mathbf{E}}_{ {\mathbf{s}}_{\tau}}\in\mathbb{R}^{L\times d}\) is the word embedding sequence of \({\mathbf{s}}_{\tau}\), and \(f\) represents a transformation function. Notice that, both \({\mathbf{E}}_{ {\mathbf{s}}_{\tau}}\) and \({\mathbf{E}}_{ {\mathbf{s}}_{\tau+1}}\) are associated with the same positional encoding \({\mathbf{U}}_{1:L}\). As a result, the model has no information to distinguish the positional difference between \(x_{\tau,j}\) and \(x_{\tau+1,j}\) for any \(j=1,\dots,L\), resulting in a sheer performance loss.
In order to avoid this failure mode, the fundamental idea is to only encode the relative positional information in the hidden states. Conceptually, the positional encoding gives the model a temporal clue or “bias” about how information should be gathered, i.e., where to attend. For the same purpose, instead of incorporating bias statically into the initial embedding, one can inject the same information into the attention score of each layer. More importantly, it is more intuitive and generalizable to define the temporal bias in a relative manner. For instance, when a query vector \(q_{\tau,i}\) attends on the key vectors \({\mathbf{k}}_{\tau,\leq i}\), it does not need to know the absolute position of each key vector to identify the temporal order of the segment. Instead, it suffices to know the relative distance between each key vector \(k_{\tau,j}\) and itself \(q_{\tau,i}\), i.e. \(i-j\). Practically, one can create a set of relative positional encodings \({\mathbf{R}}\in\mathbb{R}^{L_{\text{max}}\times d}\), where the \(i\)-th row \({\mathbf{R}}_{i}\) indicates a relative distance of \(i\) between two positions. By injecting the relative distance dynamically into the attention score, the query vector can easily distinguish the representations of \(x_{\tau,j}\) and \(x_{\tau+1,j}\) from their different distances, making the state reuse mechanism feasible. Meanwhile, we won’t lose any temporal information, as the absolute position can be recovered recursively from relative distances.
—Dai, Yang, Yang, Carbonell, Le and Salakhutdinov, "Transformer-XL: attentive language models beyond a fixed-length context" (2019). ACL 2019 (arXiv 1901.02860), Section 3.3; \(x_{\tau,j}\) is token \(j\) of segment \(\tau\). The results quoted below are from the abstract.
Its score rewrites the four terms of (4.4) one by one. The absolute encoding of the key becomes the sinusoid of the offset, \(\rho_{i-j}\) (the vector of Definition 4.2.1 at the position \(i - j\)), with a projection \(W_R\) of its own; the absolute encoding of the query becomes two learned vectors \(u\) and \(v\), the same at every position:
\[s_{ij} \;\propto\; \underbrace{q_i^{\top} k_j}_{(a)} + \underbrace{q_i^{\top} W_R\, \rho_{i-j}}_{(b)} + \underbrace{u^{\top} k_j}_{(c)} + \underbrace{v^{\top} W_R\, \rho_{i-j}}_{(d)} . \tag{4.6}\]In the words of the paper, "term \((a)\) represents content-based addressing, term \((b)\) captures a content-dependent positional bias, term \((c)\) governs a global content bias, and \((d)\) encodes a global positional bias". The score induced is \((b) + (d) = (q_i + v)^{\top} W_R\, \rho_r\), a linear readout of the clocks at the offset: a trigonometric polynomial in \(r\) whose coefficients are set by the query, applied in every layer. It is the closest additive ancestor of the rotary score. The paper reports dependencies 450% longer than those of the plain transformer and a perplexity of 18.3 on WikiText-103; the price is a second projection of every key.
The third design, of the text-to-text model T5, keeps only a number: "each “embedding” is simply a scalar that is added to the corresponding logit used for computing the attention weights", shared by all layers and different for each head.Raffel, Shazeer, Roberts and others, "Exploring the limits of transfer learning with a unified text-to-text transformer", arXiv 1910.10683, 2019, Section 2.1. In symbols, and the scaling of the dot product aside,
\[s^{(h)}_{ij} = q_i^{\top} k_j + b^{(h)}_{\beta(i - j)}, \tag{4.7}\]where \(\beta\) maps an offset to one of 32 buckets and \(b^{(h)} \in \mathbb R^{32}\) belongs to head \(h\). In the causal case, as a widely used implementation computes it, the offsets 0 to 15 have a bucket each and the other 16 buckets widen geometrically, the last holding every offset from 113 on. The score induced is a learned staircase with steps of constant width in \(\log r\), the ruler of (4.3) cut into pieces, flat in the far field and free of content. In the comparison on WikiText-103 cited above, a model with this bias kept improving for 600 tokens beyond a training length of 512, where the sinusoid managed 20, at about half the training speed.
By 2020 relative scores could be had at a price: tables, extra projections and extra products in every layer. The two schemes of 2021 cost nothing.
Rotary embeddings, and a bias with no embedding
In this section the query sits at position \(m\) and the key at position \(n\), as in the paper that introduced the scheme, because \(\mathrm{i}\) is needed for the imaginary unit; the offset is \(r = m - n\). The paper starts from the requirement of Definition 4.3.1 and asks which encodings of position meet it.
3.1 Formulation
Transformer-based language modeling usually leverages the position information of individual tokens through a self-attention mechanism. As can be observed in Equation 2, \({\boldsymbol{q}}_{m}^{\intercal}{\boldsymbol{k}}_{n}\) typically enables knowledge conveyance between tokens at different positions. In order to incorporate relative position information, we require the inner product of query \({\boldsymbol{q}}_{m}\) and key \({\boldsymbol{k}}_{n}\) to be formulated by a function \(g\), which takes only the word embeddings \({\boldsymbol{x}}_{m}\), \({\boldsymbol{x}}_{n}\), and their relative position \(m-n\) as input variables. In other words, we hope that the inner product encodes position information only in the relative form:
\[\langle f_{q}({\boldsymbol{x}}_{m},m),f_{k}({\boldsymbol{x}}_{n},n)\rangle=g({\boldsymbol{x}}_{m},{\boldsymbol{x}}_{n},m-n). \qquad (11)\]
The ultimate goal is to find an equivalent encoding mechanism to solve the functions \(f_{q}({\boldsymbol{x}}_{m},m)\) and \(f_{k}({\boldsymbol{x}}_{n},n)\) to conform the aforementioned relation.
3.2 Rotary position embedding
3.2.1 A 2D case
We begin with a simple case with a dimension \(d=2\). Under these settings, we make use of the geometric property of vectors on a 2D plane and its complex form to prove (refer Section 3.4.1 for more details) that a solution to our formulation Equation 11 is:
\[f_{q}({\boldsymbol{x}}_{m},m)=({\boldsymbol{W}}_{q}{\boldsymbol{x}}_{m})e^{im\theta} \qquad (12)\]
\[f_{k}({\boldsymbol{x}}_{n},n)=({\boldsymbol{W}}_{k}{\boldsymbol{x}}_{n})e^{in\theta}\]
\[g({\boldsymbol{x}}_{m},{\boldsymbol{x}}_{n},m-n)=\operatorname{Re}[({\boldsymbol{W}}_{q}{\boldsymbol{x}}_{m})({\boldsymbol{W}}_{k}{\boldsymbol{x}}_{n})^{*}e^{i(m-n)\theta}]\]
where \(\operatorname{Re}[\cdot]\) is the real part of a complex number and \(({\boldsymbol{W}}_{k}{\boldsymbol{x}}_{n})^{*}\) represents the conjugate complex number of \(({\boldsymbol{W}}_{k}{\boldsymbol{x}}_{n})\). \(\theta\in\mathbb{R}\) is a preset non-zero constant. We can further write \(f_{\{q,k\}}\) in a multiplication matrix:
\[f_{\{q,k\}}({\boldsymbol{x}}_{m},m)=\left(\begin{array}{cc}\cos{m\theta}&-\sin{m\theta}\\ \sin{m\theta}&\cos{m\theta}\end{array}\right)\left(\begin{array}{cc}W^{(11)}_{\{q,k\}}&W^{(12)}_{\{q,k\}}\\ W^{(21)}_{\{q,k\}}&W^{(22)}_{\{q,k\}}\end{array}\right)\left(\begin{array}{cc}x^{(1)}_{m}\\ x^{(2)}_{m}\end{array}\right) \qquad (13)\]
where \((x^{(1)}_{m},x^{(2)}_{m})\) is \({\boldsymbol{x}}_{m}\) expressed in the 2D coordinates. Similarly, \(g\) can be viewed as a matrix and thus enables the solution of formulation in Section 3.1 under the 2D case. Specifically, incorporating the relative position embedding is straightforward: simply rotate the affine-transformed word embedding vector by amount of angle multiples of its position index and thus interprets the intuition behind Rotary Position Embedding.
Equation (12) is the whole idea, and three lines show why it works. Let a head have dimension two and read a vector \(q = (q_1, q_2)\) as the complex number \(z_q = q_1 + \mathrm{i}\, q_2\). The inner product of two vectors is \(\langle q, k\rangle = \mathrm{Re}(z_q \bar z_k)\). Multiplying \(z_q\) by \(e^{\mathrm{i} m\theta}\) turns the vector by the angle \(m\theta\) and leaves its length alone, and
\[\mathrm{Re}\bigl(z_q e^{\mathrm{i} m\theta}\; \overline{z_k e^{\mathrm{i} n\theta}}\bigr) \;=\; \mathrm{Re}\bigl(z_q \bar z_k\, e^{\mathrm{i}(m - n)\theta}\bigr): \tag{4.8}\]the two absolute angles cancel into their difference. Nothing is added to anything, and no term has to be learned or deleted: the scheme is multiplicative where every earlier one was additive. The general case is this one, repeated in every pair of coordinates at a different rate.
Definition 4.4.1 (rotary position embedding). Let \(\theta_t = b^{-2t/d_h}\) for \(t = 0, \dots, d_h/2 - 1\), with a base \(b\) (\(10{,}000\) in the paper), and let the coordinates of a vector \(x \in \mathbb R^{d_h}\) be paired into complex numbers \(z_0(x), \dots, z_{d_h/2-1}(x)\). The rotary embedding of \(x\) at position \(m\) is the vector \(\mathcal R_m x\) with
\[z_t(\mathcal R_m x) \;=\; z_t(x)\, e^{\mathrm{i} m\theta_t}. \tag{4.9}\]Equivalently, \(\mathcal R_m\) is the block-diagonal matrix that rotates the pair \(t\) by the angle \(m\theta_t\). It is applied to the query and to the key of every head in every layer, \(q_m \mapsto \mathcal R_m q_m\) and \(k_n \mapsto \mathcal R_n k_n\). The values and the residual stream are left alone, and there are no parameters.
Proposition 4.4.2 (the relative property). For all \(q, k \in \mathbb R^{d_h}\) and all real \(m, n\),
\[\langle \mathcal R_m q,\, \mathcal R_n k\rangle \;=\; \mathrm{Re}\sum_t z_t(q)\, \overline{z_t(k)}\; e^{\mathrm{i}(m - n)\theta_t} \;=\; \langle \mathcal R_{m-n}\, q,\, k\rangle . \tag{4.10}\]Proof. For any \(u, v \in \mathbb R^{d_h}\), \(\langle u, v\rangle = \mathrm{Re}\sum_t z_t(u)\, \overline{z_t(v)}\), by expanding \((a + \mathrm{i} b)(c - \mathrm{i} d)\) in each pair. Apply this to \(u = \mathcal R_m q\) and \(v = \mathcal R_n k\) and use (4.8) in each pair; the last expression is the same identity for \(u = \mathcal R_{m-n} q\) and \(v = k\). ∎
Corollary 4.4.3 (the score a rotary head induces). Write \(z_t(q)\,\overline{z_t(k)} = a_t e^{\mathrm{i}\varphi_t}\): \(a_t = \lvert z_t(q)\rvert\, \lvert z_t(k)\rvert\) is the product of the lengths of the two vectors in the pair \(t\), and \(\varphi_t\) is the angle from the key to the query there. Then, with \(r = m - n\),
\[\langle \mathcal R_m q,\, \mathcal R_n k\rangle \;=\; \sum_{t=0}^{d_h/2-1} a_t \cos(r\theta_t + \varphi_t), \qquad \Bigl\lvert \sum_t a_t \cos(r\theta_t + \varphi_t)\Bigr\rvert \;\le\; \sum_t a_t \;\le\; \lVert q\rVert\, \lVert k\rVert . \tag{4.11}\]At \(r = 0\) this is the ordinary dot product.
Like the readout of Transformer-XL, (4.11) is a trigonometric polynomial in the offset, but its amplitudes and phases are set by query and key together: position modulates the content match, pair by pair.
rotary positions: a query at position m = 20 and a key at position n = 15, each turned in every plane (a pair of coordinates) by its position times the plane's rate plane (pair of a head) 1 5 9 13 tokens for one turn 6.3 20 63 199 query turned by 20 rad 6.3 rad 2.0 rad 0.63 rad key turned by 15 rad 4.7 rad 1.5 rad 0.47 rad angle between them 74° 91° 29° 9.1° = (m − n) × rate |q| |k| cos(angle) 0.28 −0.01 0.88 0.99 score 2.14 of at most 4 slide both, to m = 30 and n = 25: every vector turns, the 4 angles and the score stay slide one, to m = 21: the score becomes 2.45 (the live figure needs JavaScript and WebGL; it loads about 0.7 MB of three.js and no data)
Consider four of the 32 planes of a head of gpt-pp, with periods of 6.3, 20, 63 and 199 tokens. With the example query at position 20 and key at position 15, the rotated vectors stand 74°, 91°, 29° and 9° apart. Moving both to 30 and 25 turns every vector, by 10, 3.2, 1 and 0.32 radians, while leaving those four angles, and hence the score, unchanged.
In the code. Four lines of gpt-pp do all of this. Two fill a table with the angle of every plane at every position, which the function returns as its cosines and sines; the other two are the rotation. They are rotary_table and rotate in gpt-pp's model.py.
def rotary_table(n_positions: int, d_head: int, base: float) -> tuple[Tensor, Tensor]:
"""Cosines and sines of the rotary angles, each (n_positions, d_head / 2): pair i of a head's coordinates
turns by position * base^(-2i / d_head), so the wavelengths grow geometrically from 2 pi towards 2 pi base."""
frequency = base ** (-torch.arange(0, d_head, 2, dtype=torch.float64) / d_head)
angle = torch.outer(torch.arange(n_positions, dtype=torch.float64), frequency)
return angle.cos().float(), angle.sin().float()
def rotate(x: Tensor, cos: Tensor, sin: Tensor) -> Tensor:
"""Rotary position embedding (Su et al., 2021, arXiv:2104.09864) of queries or keys x (B, H, T, dh).
The coordinates are taken in pairs (i, i + dh/2) and each pair is rotated by the angle of its position.
Rotating the query at position m and the key at position n leaves a dot product that depends on m - n
only, so position enters where it is used, in the attention score, as a relative offset, and the residual
stream carries no position vector. It is the default of open models since Llama (2023).
"""
x1, x2 = x.chunk(2, dim=-1)
return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1).to(x.dtype)
frequency holds the rates \(\theta_t\) of Definition 4.4.1 and angle the turns \(m\theta_t\), one row for each position \(m\) and one column for each pair. In rotate, x holds the queries or the keys of every head at every position, and the line that returns is (4.9), \((x_1 + \mathrm{i}\, x_2)(\cos m\theta_t + \mathrm{i} \sin m\theta_t)\) written out, the real parts first. The formula leaves out the arithmetic: the angles are computed in float64 and kept in float32, and under the mixed precision of training (Section 6), where queries and keys arrive in the 16-bit type bfloat16, the rotation is computed in float32 and .to(x.dtype) casts it back. CausalSelfAttention.position applies it to queries and keys alone, in the line q, k = rotate(q, *rot), rotate(k, *rot), after it has normalized them (Section 3); the order does not matter, since a rotation preserves lengths.
Quirk. rotate pairs coordinate \(t\) of a head with coordinate \(t + d_h/2\): x1 and x2, the real and imaginary parts of the \(z_t\) of Definition 4.4.1, are the two halves of the vector. The paper pairs neighbors, the first coordinate with the second, the third with the fourth, and so on. The two conventions differ by a fixed permutation of a head's coordinates, which the learned \(W_Q\) and \(W_K\) absorb, so nothing changes in what a head can learn; but weights trained under one are not interchangeable with an implementation of the other. The small reference implementation nanochat, as read on 14 November 2026, splits a head into the same halves and turns by minus the angle.
Theorem 4.4.4 (rotations are the only exact solution). Let \(F_m : \mathbb R^{d_h} \to \mathbb R^{d_h}\), for real \(m\), be maps with \(F_0\) the identity and \(m \mapsto F_m(x)\) continuous, such that \(\langle F_m(q), F_n(k)\rangle\) is a function of \(q\), \(k\) and \(m - n\) only. Then \(F_m = U\, \mathcal R_m U^{\top}\) for an orthogonal matrix \(U\) and the rotary matrix \(\mathcal R_m\) of some rates \(\theta_0, \dots, \theta_{d_h/2-1}\).
Proof. With \(m = n\) the hypothesis says that each \(F_m\) preserves inner products, so it is linear and an orthogonal matrix. The hypothesis then makes \(F_m^{\top} F_n\) a function of \(m - n\), from which \(F_{-n} = F_n^{\top}\) and \(F_n = F_m F_{n-m}\): a continuous one-parameter group of orthogonal matrices, \(e^{mA}\) with \(A\) skew-symmetric. In a suitable orthonormal basis \(U\), \(A\) is block-diagonal with blocks \(\begin{pmatrix}0 & -\theta_t\\ \theta_t & 0\end{pmatrix}\), whose exponential at \(m\) is the rotation by \(m\theta_t\). ∎
The basis is harmless, since \(U^{\top}\) merges into the learned \(W_Q\) and \(W_K\): the only freedom of an exact relative scheme of this kind is its list of rates. A rate may be zero, a pair that does not see position, and the configurations of 2025 and 2026 use that freedom (Section 4.6).Su et al. (2021) give the argument for two dimensions; Schenck, Reid, Jacob, Bewley, Ainslie and others, "Learning the RoPEs: better 2D and 3D position encodings with STRING", arXiv 2502.02562, 2025, Theorems 3.2 and 3.4, for positions with several coordinates, whose generators must commute.
Example 4.4.5 (the clocks of gpt-pp and of the model in the page). The setup: closed form, no training. A head of gpt-pp has dimension 64 (width 768, 12 heads), hence 32 pairs, and the base is 10,000 as in the paper: the pair \(t\) turns by \(10^{-t/8}\) radians per token. The fastest turns by one radian (57.3°) and completes a turn every 6.28 tokens; the slowest has a period of 47,117 tokens and turns by 7.8° across the whole context of 1,024. Of the 32 pairs, 14 never complete a turn inside the context; on the ladder (the same heads, a context of 512) it is 16, and in gpt-pico, the smallest model of the series and the one that runs in the page (16 pairs, a context of 256), nine of 16. About half of a head can tell offsets apart; the other half hardly moves. Small reference implementations disagree on the base: one took 10,000 and later 100,000, another 1,024 with half of the pairs left unrotated.nanochat and the modded-nanogpt speedrun, as read on 14 November 2026. gpt-pp keeps the value of the paper; Section 4.6 asks whether that is enough. ◁
The fast pairs make sharp positional patterns possible.
Proposition 4.4.6 (a head can point at any offset). Let \(q \in \mathbb R^{d_h}\) have a nonzero component in the fastest pair, and let \(r_0\) be an integer. There is a key \(k\) for which the rotary score \(\langle \mathcal R_m q, \mathcal R_n k\rangle\), as a function of the integer offset \(r = m - n\), has its only maximum at \(r = r_0\).
Proof. Take \(z_t(k) = z_t(q)\, e^{\mathrm{i} r_0\theta_t}\): the query, turned forward by \(r_0\) steps in every pair. By (4.10) the score is \(\sum_t \lvert z_t(q)\rvert^2 \cos((r - r_0)\theta_t)\), which is at most \(\lVert q\rVert^2\), with equality only if \((r - r_0)\theta_t\) is a multiple of \(2\pi\) in every pair where \(q\) is not zero. In the fastest pair \(\theta_0 = 1\), and a nonzero integer is not a multiple of \(2\pi\). ∎
If \(W_Q\) and \(W_K\) read a direction of the stream that every token shares, and \(W_K\) writes it turned by one step in each pair, every query prefers the key one position back, whatever the tokens are: a previous-token head. A pair tuned one token back in the four planes above scores 4.00 at the offset 1 and 3.49 at the offsets 0 and 2. The peak is shallow, and nine tenths of it comes from the fastest plane; the scale of the scores is what makes a head sharp.
Remark 4.4.7 (the decay that is not built in). The paper argued for the scheme by a decay of the score with distance. Take \(q = k\) with every coordinate equal to one: the score is \(2\sum_t \cos(r\theta_t)\), the kernel of Proposition 4.2.3 at the width of a head. For a head of dimension 64 it is 64.0, 61.8, 42.1, 35.7 and 17.9 at the offsets 0, 1, 10, 100 and 1,000: the logarithmic ruler of (4.3), with a ripple, 32 clocks being few. But this is the score of one pair of vectors, aligned in every plane. For independent Gaussian \(q\) and \(k\) the expected score is zero at every offset, and by Proposition 4.4.6 a head can put its maximum wherever it likes. Rotary embeddings make a decay available; they do not impose one.Su et al. (2021), Sections 3.3 and 3.4.3, for the decay; Barbero et al. (2024), Propositions 3.1 and 3.2, for the two objections.
The other scheme of 2021 gives up position vectors altogether. (Positions are \(i\) and \(j\) again.)
In the transformer model of Vaswani et al. (2017), position embeddings are added to the word embeddings at the bottom of the network. For an input subsequence of length \(L\), the attention sublayer computes the attention scores for the \(i\)th query \(\mathbf{q}_{i}\in\mathbb{R}^{1\times d}\), (\(1\leq i\leq L\)) in each head, given the first \(i\) keys \(\mathbf{K}\in\mathbb{R}^{i\times d}\), where \(d\) is the head dimension:
\[\text{softmax}(\mathbf{q}_{i}\mathbf{K}^{\top})\]These attention scores are then multiplied by the values to return the output of the attention sublayer.
When using ALiBi, we do not add position embeddings at any point in the network. The only modification we apply is after the query-key dot product, where we add a static, non-learned bias:
\[\text{softmax}(\mathbf{q}_{i}\mathbf{K}^{\top}+m\cdot[-(i-1),...,-2,-1,0]),\]where scalar \(m\) is a head-specific slope fixed before training. Figure 3 offers a visualization.
—Press, Smith and Lewis, "Train short, test long: attention with linear biases enables input length extrapolation" (2021). arXiv 2108.12409, Section 3; the results quoted below are from the abstract, Sections 2.2 and 4.1 and Figure 1.
Definition 4.4.8 (ALiBi, attention with linear biases). With no position vector anywhere, the score of head \(h\) is
\[s^{(h)}_{ij} = \frac{q_i^{\top} k_j}{\sqrt{d_h}} - m_h\, (i - j), \tag{4.12}\]with slopes fixed before training: for \(H\) heads, \(m_h = 2^{-8h/H}\) for \(h = 1, \dots, H\), which for eight heads is \(\tfrac12, \tfrac14, \dots, \tfrac1{256}\).
In the code. The slopes, and one table that holds the bias of every head for every pair of positions. They are alibi_slopes and alibi_table in gpt-pp's model.py.
def alibi_slopes(n_head: int) -> Tensor:
...
return 2.0 ** (-8.0 * torch.arange(1, n_head + 1, dtype=torch.float64) / n_head)
def alibi_table(n_positions: int, n_head: int) -> Tensor:
"""The ALiBi bias of every (query, key) pair, (H, n_positions, n_positions): -slope * (i - j) where the key
j is not after the query i, and -inf where it is (the causal mask is folded in)."""
position = torch.arange(n_positions, dtype=torch.float64)
distance = position[:, None] - position[None, :]
bias = -alibi_slopes(n_head)[:, None, None] * distance
return bias.masked_fill(distance < 0, float("-inf")).float()
alibi_slopes returns the \(m_h\) of the definition, and the entry of bias for head \(h\), query \(i\) and key \(j\) is the second term of (4.12). The formula has no causal mask and the table does: where the key comes after the query the entry is \(-\infty\), so one float32 tensor of shape \((H, n, n)\) carries the mask and the bias together. GPT.forward cuts it to the length of the sequence and hands it to every block, where CausalSelfAttention.forward passes it to the fused attention kernel of Section 3 as attn_mask, cast to the type of the queries, with the kernel's own causal flag switched off (is_causal=bias is None).
PyTorch. F.scaled_dot_product_attention (PyTorch 2.9.1) accepts a mask passed as a tensor, but FlashAttention, the first kernel in the library's order, does not, and the next in that order is memory-efficient attention; nothing reports the change, and which kernel ran was read from the library's source, not observed. Measured: at 32 sequences of 512 tokens the table cost 6% of the rate, about 193,000 tokens a second against 205,000, and no memory; the full batch of 128 sequences was not measured. Under the mixed precision of training the cast makes the table bfloat16 on this path, while the implementation with the scores written out (Section 3), the one the analyses read, adds it in float32. The release notes of PyTorch 2.10 list "Add Flash Attention 4 to sdpa" (this function), so the kernel that takes the call may differ after 2.9.1; the series did not test it.
Remark 4.4.9 (a window, and what bounded scores do to it). Exponentiating (4.12), \(a_{ij} \propto e^{q_i^{\top} k_j/\sqrt{d_h}} \cdot e^{-m_h (i - j)}\): the content weights are multiplied by a geometric window with a characteristic length of \(1/m_h\) tokens, which is 2, 4, 8, up to 256 for eight heads. The bias is defined at every distance, so nothing is out of range at any length; that is the whole argument for extrapolation. Its cost is in the same formula: in the gentlest of eight heads a token 2,048 positions back is down-weighted by \(e^{-8} \approx 3.4 \times 10^{-4}\) before content has a say. Content can answer with a large dot product unless something bounds it, and in the recipe of gpt-pp something does: a score is 12 times the cosine of the normalized query and key (QK normalization, Section 3), so two scores differ by at most 24, and in the four steepest of eight heads no key farther than 48, 96, 192 and 384 tokens can outweigh the adjacent one. The ALiBi model of Section 4.5 is trained under that constraint, and at the training length it loses nothing measurable by it.
The headline of the paper is a model of 1.3B parameters trained on 1,024 tokens which, run on 2,048, matches the perplexity of a sinusoidal model trained on 2,048. On WikiText-103, where the sinusoid stopped improving 20 tokens beyond a training length of 512 and rotary embeddings 200 tokens beyond it, ALiBi improved up to about three times the training length. Its authors attribute most of that gain to fewer tokens being predicted with a short context, not to the use of longer histories. Later biases replace the line by a logarithm or a learned function, for a window with a power-law tail.Chi, Fan, Ramadge and Rudnicky, "KERPLE: kernelized relative positional embedding for length extrapolation", NeurIPS 2022 (arXiv 2205.09921).
| scheme | where position enters | score induced, as a function of the offset \(r\) | parameters | what it left open |
|---|---|---|---|---|
| sinusoid, 2017 | a fixed vector added to the input | none guaranteed (Proposition 4.2.4) | none | offsets must be learned; no gain beyond 20 to 50 tokens past the training length |
| learned table, 2017 | a trained vector added to the input | none by construction | \(L_{\max}\, d\) | no vector beyond \(L_{\max}\); position mixed into content |
| Shaw et al., 2018 | a trained vector for each clipped offset, added to the key | \(q_i^{\top} a_{\mathrm{clip}(-r,\, c)}\), constant beyond \(c\) | \((2c + 1)\, d_h\) a layer | no term free of content; the clocks discarded |
| Transformer-XL, 2019 | the sinusoid of the offset, projected | \((q_i + v)^{\top} W_R\, \rho_r\), coefficients set by the query | a projection, two vectors | a second projection of every key |
| T5 bias, 2019 | a trained scalar for each bucket of offsets | \(b^{(h)}_{\beta(r)}\), a staircase in \(\log r\), flat beyond 128 (in code, from offset 113) | 32 a head | half the training speed; a flat far field |
| rotary, 2021 | a rotation of query and key | \(\sum_t a_t \cos(r\theta_t + \varphi_t)\), amplitudes and phases set by query and key | none | phases never seen beyond the training length |
| ALiBi, 2021 | a fixed multiple of the distance | \(-m_h\, r\), defined at every distance | none | a window that forgets |
| none, 2022 | nowhere: the mask counts | what the stack computes from the count \(1/i\) | none | no exact positional heads; slower, less stable training |
Measured on the ladder
The ladder is the set of small models of this series that differ in one component at a time: eight layers, 512 wide, eight heads of dimension 64, a context of 512 tokens and a vocabulary of 32,768, trained on the same tokens of the web corpus FineWeb-EDU. Here it carries one model for each position scheme, with everything else equal. Losses are in nats per token.
Example 4.5.1 (five schemes at the training length). The setup: the ladder model in the recipe of gpt-pp with one switch changed. Position is rotary with base 10,000 (the recipe), or a learned table, or the sinusoid of 2017, or ALiBi, or nothing. Each model is trained on the same 50,003,968 tokens in the same order, with one seed, and scored at the end on the same 2,048 held-out windows of 512 tokens. The scale to hold differences against: two seeds of the recipe differed by 0.0006 in a short run at this size, the two seeds of each ladder rung below by 0.0025 and 0.0037, and of the ten rungs of the ladder eight have their two seeds within 0.006 of each other and two have them 0.015 and 0.018 apart. A second seed of the recipe, of the learned table and of the sinusoid, on the same tokens, ended at 4.0435, 4.1511 and 4.1019, each within 0.006 of its first run and in the same order.
In the code. The two additive schemes enter the model in one function, and the sinusoid has two places in it. It is GPT.embed_tokens in gpt-pp's model.py.
def embed_tokens(self, idx: Tensor, t0: int = 0) -> Tensor:
"""Token vectors, plus position vectors for `learned` and `sinusoidal`, for tokens at positions t0, t0+1, ...
Fixed sinusoids added before an embedding norm fade as training lengthens the token rows (the norm
divides both by the rows' scale). With `pos_after_norm` they are added after the norm instead, at unit
amplitude on a unit-rms token vector, so that position keeps a third of the energy whatever the table does.
"""
x = self.embed(idx)
after = self.cfg.pos == "sinusoidal" and self.cfg.pos_after_norm
if self.cfg.pos == "learned":
x = x + self.pos_embed.weight[t0:t0 + idx.size(1)]
elif self.cfg.pos == "sinusoidal" and not after:
x = x + self.pos_table[t0:t0 + idx.size(1)]
if self.embed_norm is not None:
x = self.embed_norm(x)
if after:
x = x + self.pos_table[t0:t0 + idx.size(1)] / self.embed_std
return F.dropout(x, self.cfg.dropout, self.training) if self.cfg.dropout > 0 else x
self.embed(idx) holds the rows \(w_i\). The branch for learned is the sum \(w_i + P_i\) of Definition 4.2.5, taken before the norm: self.pos_embed.weight is the table \(P\), with block_size rows, drawn at the scale of the token embedding and trained in the same group of parameters. The two branches for sinusoidal are the sum \(w_i + p_i\) of Definition 4.2.1, placed before or after self.embed_norm, the normalization that the recipe applies to each token vector on its way into the residual stream; self.pos_table is the table of that definition times self.embed_std (the listing of Example 4.5.2 has the line), which the last branch divides out again. Rotary positions, ALiBi and no positions add nothing here.
Bug, fixed. With pos=sinusoidal alone the table is added before the norm, and it fades: the token rows grow during training and the fixed sinusoids do not, so that in the finished first run the rows had a root mean square of 5.86 against 0.707 for the sinusoids, and the share of position in the energy entering the norm had fallen from 33% to 1.4%. That run ended at 4.1777, 0.133 behind rotary and behind the learned table (4.1572). The fix is the switch pos_after_norm, which adds the sinusoids after the norm at unit amplitude; it is off by default, so that the first run can still be reproduced, and the sinusoid's row below therefore changes two fields of the configuration where the other rows change one. Every sinusoid number of this series is measured with the switch on: here 4.1050 and 4.1019 in the two seeds, 0.060 and 0.058 behind rotary and ahead of the table.
| position scheme | position parameters | held-out loss | against rotary |
|---|---|---|---|
| learned table | 262,144 | 4.1572 | +0.112 |
| sinusoid, 2017 | 0 | 4.1050 | +0.060 |
| rotary, base 10,000 (the recipe) | 0 | 4.0451 | |
| ALiBi | 0 | 4.0471 | +0.002 |
| none | 0 | 4.3093 | +0.264 |
Rotary and ALiBi are 0.0020 apart, less than two seeds of one configuration can differ by: one run each cannot rank them. The sinusoid is 0.060 behind and the learned table 0.112. The sinusoid is thus 0.052 ahead of the table, nearly three times the distance between the two most distant seeds we have (0.018), where the 2017 paper found "nearly identical results". The model with no positions is 0.264 behind, and was behind from the first evaluation (by 0.435 after 10M tokens): the gap narrows and does not close in this budget. Rotary and ALiBi level at the front, then the sinusoid, the learned table and no positions: this is the order of the larger models of Section 4.1, which had no rotary model. The bounded scores of Remark 4.4.9 cost ALiBi nothing that one run can show.
The same switch is thrown once more on the ladder proper, where each rung changes one more component of the 2019 recipe and is trained on 100M tokens. The rung with a learned table scores 4.1346 and the rung with rotary positions 4.0242, means of two seeds that differ by 0.0025 and 0.0037: a step of 0.110, against 0.112 for the same swap inside the recipe of gpt-pp, and six times the largest difference between two seeds on the ladder (0.018). ◁
Example 4.5.2 (trained on 512 tokens, read out to 2,048). The setup: the same five models, run without further training on 1,536 held-out windows of 2,048 tokens, four times the training length, with the loss recorded at every position. The learned table has no row beyond 512 and cannot be run there. The sinusoidal model receives vectors it has never seen; the rotary model receives nothing new at its input, but half of its 32 pairs turn through angles that training never showed; the bias of ALiBi continues along its line; the model with no positions counts on. Each model is held against its own loss over positions 257 to 512, the second half of the training length, because the first tokens of a window have little context and raise any average that includes them.
In the code. Nothing is retrained or converted for the longer windows: the position tables are computed, and the model computes them again when a longer sequence arrives. The listing has the end of the constructor, which registers them, and the method that fills them: GPT.__init__ and GPT.extend_positions in gpt-pp's model.py.
def __init__(self, cfg: ModelConfig):
...
# Position tables are computed, not learned: buffers kept out of the checkpoint, extended on demand.
self.n_positions = 0
for name in ("pos_table", "rope_cos", "rope_sin", "alibi"):
self.register_buffer(name, None, persistent=False)
self.extend_positions(cfg.block_size)
...
def extend_positions(self, n: int) -> None:
"""Make the position tables cover n positions. A model with sinusoidal, rotary, ALiBi or no positions
accepts sequences longer than it was trained on, which is how length generalization is measured; a
learned table cannot be extended."""
if n <= self.n_positions:
return
cfg, device = self.cfg, self.embed.weight.device
if cfg.pos == "learned" and n > cfg.block_size:
raise ValueError(f"learned positions end at block_size {cfg.block_size}; got a sequence of {n}")
if cfg.pos == "sinusoidal":
# The 2017 model added unit-amplitude sinusoids to unit-variance embeddings; keep that ratio.
self.pos_table = (sinusoidal_table(n, cfg.d_model) * self.embed_std).to(device)
elif cfg.pos == "rope":
cos, sin = rotary_table(n, cfg.d_head, cfg.rope_base)
self.rope_cos, self.rope_sin = cos.to(device), sin.to(device)
elif cfg.pos == "alibi":
self.alibi = alibi_table(n, cfg.n_head).to(device)
self.n_positions = n
n is the length of the sequence that arrives, 2,048 in this example against a block_size of 512, and GPT.forward makes the call for every batch. Each branch calls the function listed with its scheme in Sections 4.2 and 4.4. The five values of cfg.pos are the five rows of the tables of this section; the relative schemes of Section 4.3 and the rescaled or partly unrotated rotations of Section 4.6 are described in this part and not implemented in gpt-pp.
Quirk. The sinusoidal, rotary and ALiBi tables are buffers registered with persistent=False: a checkpoint does not contain them, and they are built again when a model is loaded and extended when a longer sequence arrives. The sinusoidal table is multiplied by embed_std, the standard deviation at which the token embedding is drawn (1 for the untied embedding of the recipe of gpt-pp, 0.02 for a tied embedding or the 2019 recipe), to keep the ratio of the 2017 model, sinusoids of unit amplitude on embeddings of unit variance. A learned table is a parameter with block_size rows: for a longer sequence the model raises ValueError, which is the "no row" of the table below.
five models of the ladder, alike but for the position scheme, trained on windows of 512 tokens, scored on held-out text in windows of 2,048: loss in bits per token, by position positions 257 to 512 513 to 1,024 1,025 to 2,048 change past 512 final rotary 5.78 5.86 6.20 +0.42, does not hold 5.84 learned 5.95 no position 513: its table ends 6.00 sinusoidal 5.87 8.01 8.66 +2.79, does not hold 5.92 ALiBi 5.78 5.77 5.76 −0.02, holds 5.84 none 6.19 7.22 9.70 +3.52, does not hold 6.22 sinusoidal: here the sinusoids are added after the embedding's normalization; a first run, with the sinusoids added before the embedding's normalization, where they fade as the token vectors grow, ended at 6.03 bits per token: a fading signal, not drawn the heads of rotary, reading from positions 257 to 512, put 39 % of their attention on the 8 nearest tokens; from 1,537 to 2,048, 53 % on tokens 512 or more back, which training never showed (one head: 83 %). (holds: within 0.05 bits per token of its loss at 257 to 512; one seed each) (the live figure needs JavaScript and loads about 36 kB of data)
| position scheme | positions 1 to 512 | 257 to 512 | 513 to 1,024 | 1,025 to 2,048 |
|---|---|---|---|---|
| learned table | 4.191 | 4.122 | no row | no row |
| sinusoid, 2017 | 4.141 | 4.069 | 5.555 | 6.001 |
| rotary, base 10,000 (the recipe) | 4.081 | 4.007 | 4.063 | 4.299 |
| ALiBi | 4.083 | 4.009 | 3.997 | 3.995 |
| none | 4.343 | 4.288 | 5.007 | 6.725 |
Only ALiBi holds: 4.009 over the second half of the training length, then 3.997 and 3.995. The rotary model is level just past the boundary (4.003 over the first 32 positions), stands above its level from about position 700, and then climbs steadily: over positions 1,025 to 2,048 it is 0.29 above its level, or 0.42 bits per token, and over the last 256 positions 0.432. The sinusoid jumps at the boundary itself, by 0.746 in the first 32 positions and 1.319 over the first 256, and goes on rising more slowly, to 2.057 above its level over the last 256. The model with no positions holds for 32 positions, then rises for the rest of the window and ends 2.865 above its level, the worst of the four. For the three schemes they share, this is the order of the published comparison on WikiText-103. The last place of the model with no positions is the surprise: on small algorithmic tasks that scheme generalizes in length better than the others (Section 4.6).
The heads say why. Read from positions 1,537 to 2,048, a head that weighed every token alike would put about 71% of its attention on tokens 512 or more back, which training never showed. The rotary heads put 53% there. Inside the training length their weights fall with distance, to a seventh of the uniform level at 384 to 511 tokens back; past the offset 512 the fall stops and the weights return to three quarters of that level, so that the far tokens, 1,536 of them at the end of the window, draw attention from the near ones, and the share of the offsets 1 to 3 falls from 22% to 12%. The crowd of far tokens grows with the position, which fits the slow and steady rise of the loss. The sinusoidal model and the model with no positions lose their short-range heads altogether (the share of the offsets 1 to 3 falls from 15% and 13% to 0.2%) and read about as a uniform head would: the sinusoidal heads put 75% of their attention 512 or more tokens back, every one of the 64 more than half, and the heads with no positions 64%. One reading fits both. Each carries absolute positions, and each meets positions it has never seen: the sinusoid as vectors at its input, the model with no positions as the count of Proposition 4.1.3, which no training window took beyond 512. The ALiBi heads spend 2% of their attention beyond 512, and their share of the offsets 1 to 3 stays at 21%. Trained on 512 tokens and read out to 2,048, only ALiBi holds its loss, and it holds it by not reading what was added. ◁
Example 4.5.3 (what the heads see inside the training length). The setup: the same models on 64 held-out windows; for each of the 64 heads, the mean weight on one token as a function of how far back the token is, for readers at positions 257 to 512. Every such reader has at least 257 keys, so a head that reads every token alike gives a level line and a falling curve is a preference for near tokens. Half of the attention of the median head falls within about 18 tokens with the learned table, 58 with the sinusoid, the farthest of the five, 21 with rotary, 16 with ALiBi and 43 with no positions (192 for a uniform head); the measure is coarse and does not tell the table, rotary and ALiBi apart. The sharp heads do. At least half of the attention falls on the offsets 1 to 3 in seven heads with ALiBi, four with rotary, four with the sinusoid, one with the table and one with no positions. Two rotary heads and one ALiBi head put more than half on the previous token alone, the sharpest 0.58; no head of the learned or the sinusoidal model does (0.42 and 0.46 at most). With no positions no head puts more than 0.29 on any one of the offsets 1 to 3, which is Remark 4.1.4 on real text, and all 16 heads of the first two layers read every token alike to within a tenth of their attention, which is how the head of Proposition 4.1.3 reads. The ALiBi heads sit inside their windows: half of their attention falls within 1.0, 2.5, 5.5, 11, 22, 57, 92 and 124 tokens for the eight windows. ◁
What does this say about the recipe? gpt-pp uses rotary positions, and the measurement does not single them out: at this size and budget ALiBi did as well at the training length and better beyond it. Two limits stand against reading more into that. Each number is one seed of 50M tokens on a model of eight layers. And the test past the training length rewards a model for ignoring the tokens it was not trained to reach; it never asks it to find one of them. The recipe took the rotation as the scheme of the open models (Section 4.6), not on the strength of this comparison. What the ladder does establish is that a learned table costs 0.11 nats per token against the rotation in either recipe and 0.05 against the sinusoid, and that a model with no positions is last at the training length and at the far end of the long window.
What the frequencies are for, longer contexts, and what is open
Rotary embeddings became the default of open language models in 2023. The questions since then are about the rates: which of them a trained model uses, and what to do with them when the context grows.
Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust ‘positional’ attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.
—Barbero, Vitvitskyi, Perivolaropoulos, Pascanu and Veličković, "Round and round we go! What makes rotary positional encodings useful?" (2024). arXiv 2410.06205, abstract; the perplexities below are from their Table 2 and the limit from their Theorem 6.1.
Both findings can be read off (4.11). A positional head needs pairs in which one step is a visible angle. A head that matches content wants a score that does not move with the offset, and a pair whose period is much longer than the context provides one, since \(a_t \cos(r\theta_t + \varphi_t) \approx a_t \cos\varphi_t\) while \(r\theta_t\) is small: the slow pairs are being used as plain dot-product attention. The use has a limit, which the paper proves for a head of one pair: a pair with a nonzero rate turns in the end, and in a long enough context it cannot hold a head on its target at every distance. The remedy it tests is to stop turning the slowest pairs. On a Gemma of 2B parameters, removing the rotation from the slowest quarter of the pairs gave a perplexity of 4.4414 on the first of its two validation sets, against 4.4627 with every pair rotated and 4.8594 with none. The modified scheme is still exactly relative: Proposition 4.4.2 allows a rate of zero.
The configurations of released models moved the same way.
| model | date | position scheme | base | context, tokens |
|---|---|---|---|---|
| GPT-2 | February 2019 | learned table | 1,024 | |
| Gemma 3 27B | March 2025 | rotary, global and local layers at different bases | 1,000,000 and 10,000 | 131,072 |
| Llama 4 Scout | April 2025 | rotary in three of every four layers, nothing in the fourth | 500,000 | 10,485,760 |
| Qwen3.5 27B | February 2026 | rotary on a quarter of the dimensions, in its attention layers | 10,000,000 | 262,144 |
| Gemma 4 | March 2026 | global layers: the fastest quarter of the rates kept, the rest set to zero; local layers: rotary | 1,000,000 and 10,000 | 262,144 |
| Mistral Large 4 | September 2026 | rotary with YaRN, nothing in every fourth layer | 10,000 | 1,048,576 |
| gpt-pp | November 2026 | rotary on every pair, in every layer | 10,000 | 1,024 |
Three departures from the paper of 2021 are visible. Not every channel is rotated: in most families released since 2025 the rotation covers a quarter to a half of the dimensions of query and key, and the rest is a content channel. Not every layer is rotated: three families leave one layer in four with no position signal, after a study which found that such layers retrieve by content while the rotated layers supply recency.Yang, Venkitesh, Talupuru and others, "Rope to Nope and back again: a new hybrid attention strategy", arXiv 2501.18795, 2025. And the base grew with the context, to between \(10^5\) and \(10^9\). Of 24 open models and families released since January 2025 whose configurations were read for this series, 21 rotate in at least some layers.
Why should the base grow with the context? One argument gives the curve of Remark 4.4.7 a meaning: half of it, \(B_r = \sum_t \cos(r\theta_t)\), is proportional to the expected advantage, at the offset \(r\), of a key that resembles the query over an unrelated one, and where it is negative a head that matches by similarity prefers the unrelated token on average.Men, Xu, Wang and others, "Base of RoPE bounds context length", arXiv 2405.14591, 2024, Theorem 1 and Table 2, which gives the smallest base that keeps \(B_r\) non-negative up to a context length. For the 32 pairs of gpt-pp at base 10,000, \(B_r\) is negative at five offsets up to 1,024, the first of them 725, and never below \(-1.5\), where \(B_0 = 32\); on the ladder and in gpt-pico it stays positive over the whole context. By this criterion, an average over idealized vectors and not a measurement, the base of gpt-pp is marginally small for its context. Measured on the 12-layer model at that context (100M tokens, one seed each), a base of 100,000 ends level with the paper's 10,000, at 3.649 against 3.654 nats a token, and a base of 1,000 ends 0.019 behind, at 3.673: too small a base costs, and the margin of the criterion does not show.
When a trained model is run beyond its training length \(L\), at \(sL\), its rotary pairs meet phases they have not seen: a pair whose period is shorter than \(L\) showed its whole circle in training, a pair with a longer period only a part.Liu, Yan, Zhang, An, Qiu and Lin, "Scaling laws of RoPE-based extrapolation", ICLR 2024 (arXiv 2310.05209), Lemma 1. Every remedy is a change of clock in (4.9). Position interpolation divides the positions by \(s\), so that no phase leaves the trained range, extrapolation being feared to "lead to catastrophically high attention scores that completely ruin the self-attention mechanism".Chen, Wong, Chen and Tian, "Extending context window of large language models via positional interpolation", arXiv 2306.15595, 2023, abstract. The objection came within months, and it is about the fast pairs:
If we look at RoPE only from an information encoding perspective, it was shown in [36], using Neural Tangent Kernel (NTK) theory, that deep neural networks have trouble learning high frequency information if the input dimension is low and the corresponding embeddings lack high frequency components. Here we can see the similarities: a token’s positional information is one-dimensional, and RoPE expands it to an n-dimensional complex vector embedding.
RoPE closely resembles Fourier Features [36] in many aspects, as it is possible to define RoPE as a special 1D case of a Fourier Feature. Stretching the RoPE embeddings indiscriminately results in the loss of important high frequency details which the network needs in order to resolve tokens that are both very similar and very close together (the rotation describing the smallest distance needs to not be too small for the network to be able to detect it).
—Peng, Quesnelle, Fan and Shippole, "YaRN: efficient context window extension of large language models" (2023). arXiv 2309.00071, Section 3.1 of the second version, of November 2023 (the third, of February 2026, words this section differently); reference [36] is Tancik and others (2020), on Fourier features. The rule described next is their Section 3.2, with thresholds of one and 32 periods for the Llama models.
The repairs keep the fast pairs and slow the slow ones. Raising the base from \(b\) to \(b\, s^{d_h/(d_h - 2)}\), a rule known as NTK-aware scaling (NTK abbreviates neural tangent kernel), slows the slowest pair by exactly \(s\) and leaves the fastest alone. YaRN divides the rate by \(s\) in the pairs with less than one period inside \(L\), keeps the pairs with more than 32, mixes the two in between, and sharpens the scores by a temperature.
Whether stretched clocks give a context that is used is disputed. The strongest objection, of December 2025:
Taken together, Observations 1 and 2 imply that providing explicit positional information with PE is a key component for effective LM training, but is also a fundamental barrier to long-context generalization. This raises a natural question: is it possible to harness the inductive bias from positional embeddings exclusively during training? We answer in the affirmative. In this section, we demonstrate that it is possible to drop all positional embeddings from a pretrained transformer and quickly recover the model’s in-context capabilities with a brief recalibration phase. Most notably, this simple new procedure (termed DroPE) unlocks strong zero-shot long context generalization to unseen sequence lengths, far beyond highly-tuned RoPE extensions and prior alternative architectures.
Observation 3. Positional embeddings can be removed after pretraining, allowing LMs to generalize zero-shot to unseen sequence lengths without compromising their in-context performance after short recalibration on a fraction of the training tokens at the original context size.
—Gelberg, Eguchi, Akiba and Cetin, "Extending the context of pretrained LLMs by dropping their positional embeddings" (2025). arXiv 2512.12167, Section 5; the retrieval rates are from their Table 1, 500 trials each on three tasks.
Its test is retrieval at twice the training context, on models of half a billion parameters. Plain rotary embeddings and position interpolation retrieve nothing (0.0% on each of three tasks); NTK-aware scaling retrieves 16.5% to 21.1%, YaRN 0.5% to 17.8%, ALiBi 0.0% to 5.2%, a model trained with no positions 9.2% to 36.2%, and the model whose rotation was dropped 23.3% to 41.6%. In this account position is scaffolding for training. It is one paper, and its model with no positions fares better than ours of Example 4.5.2, a tenth of the size and scored on loss.
A different line asks why position should be a count of tokens at all. Contextual position encoding lets a head measure the distance to a key as a sum of gates that it computes on the tokens between, so that it can count sentences where it now counts tokens, and attend, in the words of the paper, to "the \(i\)-th sentence".Golovneva, Wang, Weston and Sukhbaatar, "Contextual position encoding: learning to count what's important", arXiv 2405.18719, 2024, abstract.
What is open in November 2026, with the evidence on each side:
- Is no position good enough? A model with no positions is the best of five schemes on small algorithmic tasks in the study cited in Section 4.1, and in a study of September 2026 the best at recalling over distances unseen in training, where the rotary model is the worst. It is slightly behind in perplexity at every size, and at larger scale it shows loss spikes, which rotating a few channels or normalizing queries and keys (as the recipe of gpt-pp does) removes. In our own runs it was 0.26 nats per token behind at the training length and the worst at the far end of the long window.Nevermann and Gros, "Distance generalization in transformers: why bother with positional encoding?", arXiv 2609.11913, 2026. Khan, Gummadi, Gupta and Ravichander, "Fractional rotation, full potential? Investigating performance and convergence of partial RoPE", EMNLP 2026 (arXiv 2603.11611).
- Why does a rotary model fail beyond its training length? Phases never seen in the slow pairs; content heads displaced when those pairs are compressed; and, in an account of 2026, the failure of the token that heads use as a sink for attention they do not need. In our rotary model the visible symptom is a weight that stops falling with distance where training stopped (Example 4.5.2).Wertheimer, Zhang, Liu, Yin and Wang, "Frayed RoPE and long inputs: a geometric perspective", ICLR 2026 (arXiv 2603.18017).
No paper of 2026 that we found replaces rotary embeddings in a released large model: what changed is how much is rotated, in which layers, at which base, and what the rotary layers are mixed with. A survey of August 2026 concludes that being able to compute a position beyond the training length is not the same as using a long context well,Li, "Position encoding in transformers: from absolute and relative methods to rotary position embeddings and long-context scaling", arXiv 2608.10021, 2026, abstract. which is also what the ladder measured. The block around the score has changed as much as the position inside it, and it is the subject of Section 5.