ALiBi: Biasing Attention by Distance to Solve the Length Extrapolation Problem

The transformer architecture introduced by Vaswani et al. in 2017 left one question unanswered: how does a model generalize to sequences longer at inference time than it saw during training? Positional embeddings, the standard mechanism for telling a transformer where each token sits in a sequence, carry an implicit assumption that the positions seen during training are the same ones the model will encounter at test time. When inference sequences are longer, that assumption fails. Sinusoidal position embeddings, the original method from the 2017 paper, degrade sharply when tested beyond their training length. Rotary position embeddings (RoPE) improve on sinusoidal embeddings but still show degraded perplexity past the training cutoff. The T5 relative bias extrapolates better than either but is computationally expensive. Attention with Linear Biases (ALiBi), introduced in 'Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation' (arXiv:2108.12409) by Ofir Press (University of Washington and Facebook AI Research), Noah A. Smith (University of Washington and Allen Institute for AI), and Mike Lewis (Facebook AI Research), and accepted at ICLR 2022, presents a different solution: drop positional embeddings entirely and instead subtract a distance-proportional penalty from each attention score. The paper reports that a 1.3 billion parameter ALiBi model trained on 1024-token sequences matches the perplexity of a sinusoidal model trained on 2048-token sequences, while training 11% faster and using 11% less memory.

Why standard position methods fail to extrapolate

The core problem with positional embeddings is that they are defined over a fixed range of positions. A sinusoidal model trained with sequences of length L has never received a training gradient for position L+1 or beyond; those positions exist mathematically but are meaningless to the model. The paper measures this failure directly on WikiText-103: a sinusoidal model trained at L=512 sees perplexity rise sharply the moment evaluation sequences exceed 512 tokens. Rotary position embeddings, which encode position through a rotation in the complex plane applied to query and key vectors, partially alleviate the problem because their relative-distance structure is more generalizable than absolute positions. The T5 bias, a learned scalar per relative distance bin added to each attention score, achieves better extrapolation still, because relative distances are naturally open-ended. The catch is that the T5 bias requires additional learned parameters per attention head and is considerably slower and more memory-intensive than the sinusoidal baseline. ALiBi eliminates all these tradeoffs by making the distance penalty fixed rather than learned.

How ALiBi works: fixed slopes on attention scores

ALiBi removes positional embeddings from the model entirely. In their place, it adds a scalar penalty directly to each query-key attention score before the softmax. The penalty is m times the distance between the query position and the key position, where m is a fixed head-specific slope set before training and not updated by the optimizer. A query at position 10 attending to a key at position 7 receives a penalty of m * 3; attending to a key at position 9 receives m * 1. Nearby tokens are penalized less and attend more strongly; distant tokens are suppressed proportionally. The slope m is different for each attention head and is set by a fixed geometric sequence: for a model with n heads, the slopes are 2^(-8/n), 2^(-16/n), and so on up to 2^(-8), evenly spaced in that geometric range. Because the slopes are scalars fixed before training, ALiBi adds zero new parameters and requires no extra runtime relative to a sinusoidal model trained on the same length. The memory overhead is 0 to 0.7% compared to the sinusoidal baseline at the same training length. The method can be implemented by modifying a few lines of standard attention code.

• No position embeddings: ALiBi removes them entirely rather than replacing them with another embedding type.
• Additive penalty: the distance term is added to the query-key dot product before softmax, not after.
• Fixed slopes: head-specific slopes follow a geometric sequence set before training, with no additional parameters.
• Recency bias: the penalty grows with distance, so the model develops an inductive bias toward nearby context that generalizes across lengths.
• Implementation cost: changing a few lines of attention code; zero new parameters; 0 to 0.7% memory overhead.

Benchmark results on WikiText-103 and the 11 percent efficiency gain

The paper evaluates ALiBi against sinusoidal embeddings, rotary position embeddings, and the T5 bias on WikiText-103, a standard language modeling benchmark. Figure 1 of the paper plots perplexity (lower is better) against evaluation sequence length for models trained at L=512 and L=1024. Sinusoidal and RoPE models show rapidly increasing perplexity as evaluation length exceeds training length. The T5 bias holds up better but is slower and requires extra memory. ALiBi shows no such degradation: perplexity continues to improve as evaluation length increases past the training cutoff, up to the point described in the next section. The headline comparison is the 1.3 billion parameter model trained at L=1024 with ALiBi. When evaluated at L=2048 (twice its training length), it achieves the same perplexity as a sinusoidal 1.3B model trained directly at L=2048. The ALiBi model also trains 11% faster and uses 11% less memory than that sinusoidal model trained at the longer length, because it was able to train on shorter sequences throughout and still generalize at inference time.

How far beyond training length can ALiBi reach

The paper finds that ALiBi performance peaks at approximately twice the training sequence length. A model trained at L=512 reaches its best perplexity around L=1024; a model trained at L=1024 reaches its best perplexity around L=2048. Beyond that peak, perplexity rises slightly but does not collapse. The 1.3B model trained on L=1024 maintains strong perplexity at evaluation lengths up to at least 10,000 tokens. The recency bias induced by the distance penalty is the likely explanation: the model learns to heavily weight nearby context, and that pattern continues to be useful regardless of how long the total sequence is. A key practical consequence is that ALiBi lets teams train on shorter sequences (which are cheaper, since attention scales quadratically with length) and then deploy on longer sequences at inference time, with predictable behavior. The paper compares this directly: training a 1.3B model at L=1024 with ALiBi costs roughly the same as training at L=1024 with sinusoidal embeddings, yet it matches the quality of a sinusoidal model that paid the full cost of training at L=2048.

Adoption in BLOOM and the MPT model series

ALiBi moved into large-scale production use shortly after the ICLR 2022 publication. The BigScience research workshop chose ALiBi as the position method for BLOOM, the 176 billion parameter multilingual open-access model released in July 2022. BLOOM was trained on the Jean Zay supercomputer in France, coordinated by Hugging Face, and spans 46 natural languages and 13 programming languages; ALiBi was selected partly for its ability to handle the varied sequence lengths in a multilingual corpus without the overhead of a learned relative bias. MosaicML adopted ALiBi for the MPT (MosaicML Pretrained Transformer) model series. MPT-7B, released in 2023 under the Apache 2.0 license and trained on 1 trillion tokens, uses ALiBi combined with FlashAttention. MosaicML reported that MPT models trained with ALiBi on sequences of up to 65,000 tokens could handle evaluation sequences of up to 84,000 tokens. MPT-7B-StoryWriter-65k+ was fine-tuned at a 65,000-token context window specifically to demonstrate long-document generation enabled by ALiBi's extrapolation property. By the mid-2020s, most frontier models had moved to RoPE variants with post-hoc context extension techniques such as YaRN and NTK-aware interpolation, which allow RoPE models to be extended without retraining. ALiBi lacks a direct analog to those fine-tuning approaches, which is the primary reason it was displaced at the frontier. It remains a clean and practical choice for training long-context models from scratch, where its zero-parameter cost and predictable extrapolation behavior are genuine advantages. PromptingIndex covers ALiBi alongside RoPE, grouped-query attention, multi-head latent attention, and FlashAttention as part of its series on position encoding and attention efficiency.