Rotary Position Embedding: How RoPE Powers Llama, Mistral, and Nearly Every Modern Open LLM

When the Transformer was introduced in 2017, it included a sinusoidal positional encoding scheme: sine and cosine functions applied to each position index, then added to the input embeddings before they entered the attention layers. This approach works, but it carries a fundamental limitation. The positional signal is added to the embedding and then effectively discarded during the attention dot product, which means the model reasons about absolute positions rather than relative ones. Two tokens at positions 5 and 10 carry different absolute encodings, but what attention actually needs to capture is that they are 5 positions apart, regardless of where they sit in the sequence. Rotary Position Embedding (RoPE), introduced by Jianlin Su in April 2021 (arXiv:2104.09864), solves this by encoding position directly into the attention operation through rotation rather than addition.

The paper and its origin

The formal RoPE paper, titled 'RoFormer: Enhanced Transformer with Rotary Position Embedding,' was submitted to arXiv on April 20, 2021. The authors are Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Su, a researcher known in Chinese NLP circles, originally published the core RoPE idea on his blog kexue.fm in two posts earlier in 2021 before formalizing the method in the preprint. The technique might have remained obscure in English-language research without a second publication: EleutherAI's team (Stella Biderman, Sid Black, Charles Foster, Leo Gao, Eric Hallahan, Horace He, Ben Wang, and Phil Wang) wrote a blog post titled 'Rotary Embeddings: A Relative Revolution' also in 2021, which brought RoPE to wider attention. EleutherAI implemented RoPE in their GPT-NeoX and Mesh Transformer JAX codebases in 2022, and Meta adopted it for Llama 1 in February 2023, setting in motion a chain of adoption across nearly every open-weight model family released since.

How rotation encodes position in query and key vectors

RoPE works by treating consecutive pairs of elements in each query or key vector as coordinates of a complex number, then rotating each pair by an angle proportional to the token's position in the sequence. For a query vector at position m, each element pair (q_{2j}, q_{2j+1}) is rotated by the angle m times theta_j, where theta_j equals 10000 raised to the power of negative 2j divided by d, and d is the head dimension. When j equals 0, the rotation frequency is 1, changing rapidly from token to token. When j equals d/2 minus 1, the frequency is 10000 to the power of negative 1, equal to 0.0001, changing very slowly across long distances. This mirrors the structure of sinusoidal embeddings from the original Transformer paper but applies multiplicatively to query and key vectors rather than additively to token embeddings.

The mathematical property that makes RoPE useful is this: when you compute the dot product between a rotated query at position m and a rotated key at position n, the result depends on the difference (m minus n) and not on m and n individually. The absolute positions cancel in the dot product, leaving only the relative position. The model therefore learns how far apart tokens are without being anchored to specific positions in an absolute sequence, which also means RoPE models generalize better to variable sequence lengths than models using learned absolute position embeddings.

• RoPE applies only to query and key vectors, not to value vectors. The value vectors carry content, not positional signal.
• The base frequency theta equals 10000 in the original paper. This value determines the maximum wavelength of the lowest-frequency dimension, which governs the longest position differences the model can reliably distinguish.
• No extra parameters are required: the rotation angles are computed deterministically from position index and dimension index. RoPE adds zero learnable parameters to the model.
• Attention scores decay naturally with token distance because rotating two vectors by increasingly different angles reduces their dot product, giving distant tokens less influence without requiring explicit masking or bias terms.
• RoPE is compatible with linear attention variants, unlike T5 relative bias, which requires constructing the full N times N attention matrix.

Adoption across the major open-weight model families

GPT-NeoX 20B, released by EleutherAI in April 2022, was one of the first large open models to use RoPE, giving the community an early reference implementation. Meta's Llama 1, released in February 2023, used RoPE with the original base frequency of 10000 and a 4096-token context window. Llama 2, released in July 2023, kept the same RoPE configuration. Mistral AI's Mistral 7B, released in September 2023, uses RoPE with a native 8192-token context. Code Llama, released by Meta in August 2023, used a modified RoPE with NTK-aware frequency scaling to support 16384-token and 100000-token contexts. Llama 3, released in April 2024, substantially increased the base frequency from 10000 to 500000 to support a 128000-token context window. Google's Gemma models and Alibaba's Qwen series also use RoPE, making it the de facto standard for rotary position encoding across the open-weight ecosystem.

The context extension problem and three solutions

When a model trained with RoPE at 4096 tokens processes an 8192-token input, it generates rotation angles that were never seen during training. The low-frequency RoPE dimensions, which track long-range relationships, have not completed even one full rotation within the training context length. At inference time beyond that length, the angles enter uncharted territory and perplexity spikes sharply. Three approaches address this problem.

Position Interpolation (Chen et al., arXiv:2306.15595, June 2023) scales all position indices down proportionally so they stay within the trained range, then fine-tunes the model on a small amount of long-context data. The method works but compresses all RoPE frequencies uniformly, which hurts short-range token relationships because high-frequency dimensions get over-compressed along with the low-frequency ones. NTK-aware scaling, developed in community research by 'bloc97' in 2023, changes the base frequency parameter rather than the position indices. Raising the base from 10000 to a larger value spreads out the rotation spectrum, leaving high-frequency dimensions mostly intact while extending the range of low-frequency ones. Code Llama used a version of this approach. YaRN (Yet another RoPE extensioN method), published by Bowen Peng, Jeffrey Quesnelle, and Enrico Shippole of Nous Research and EleutherAI (arXiv:2309.00071, September 2023), combines NTK-by-parts interpolation with a temperature correction that compensates for the entropy shift in attention distributions caused by context extension. The YaRN paper reports that its approach requires 10 times fewer training tokens and 2.5 times fewer training steps than Position Interpolation to reach equivalent long-context quality. The authors extended Llama 2 7B to 64000-token context with scale factor s equal to 16, and to 128000-token context with scale factor s equal to 32.

The base frequency theta and the push toward million-token contexts

The original theta of 10000 was calibrated for 4096-token contexts. As the field pushed toward longer contexts, a pattern emerged: raising theta allows the low-frequency RoPE dimensions to complete fewer full rotations within a given context length, which keeps more positions in-distribution relative to training. Llama 3's theta of 500000 reflects this: at 128000 tokens, the lowest-frequency dimension completes only about 0.013 full rotations, compared to 1.3 full rotations that would occur with theta of 10000. LongRoPE (arXiv:2402.13753, 2024) extended this reasoning further, using non-uniform rescaling factors across different RoPE dimensions to reach context lengths beyond 2 million tokens, with the key observation that dimensions differ in how far they can be extrapolated before quality degrades.

For practitioners fine-tuning models for long-context use, the RoPE theta value matters as much as the training context length. A model trained at 8192 tokens with theta of 10000 will degrade more sharply beyond that length than a model trained at the same context length with a much higher theta, because the higher base frequency distributes the rotation spectrum across a wider position range and leaves less of the frequency space out-of-distribution. PromptingIndex covers RoPE alongside related inference-time techniques including speculative decoding and grouped query attention, both of which are architectural features that interact with the same long-context serving challenges that motivated the RoPE extensions described here.