Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

ALiBi: Attention with Linear Biases

3 minute read

Published:

TL;DR: ALiBi (Attention with Linear Biases) adds no positional embeddings to token vectors. Instead, it subtracts a fixed linear penalty — proportional to token distance — from each attention score. Zero learned parameters, excellent length extrapolation.

The Simplest PE That Works

Press, Smith, and Lewis (2022) asked: do we even need position vectors? What if we just penalise attending to far-away tokens directly in the attention scores?

The idea: tokens that are far apart should pay a cost for attending to each other. Nearby tokens are cheap to attend to; distant ones are expensive.

They achieve this with a single number: a negative slope m per attention head.

The Formula

Attention score(i, j) = q_i · k_j / √d_k − m × |i − j|

That’s it. The only change to standard attention is subtracting m × |i − j| from each score before the softmax. No PE vectors are added to embeddings at all.

Different attention heads use different slopes m, following a geometric sequence: {1/2, 1/4, 1/8, …, 2^(-h)} for h heads. Some heads focus locally (large m = steep penalty); others look further (small m = gentle penalty).

Raw attention scores − Linear bias (m × |i−j|) 5.2 4.1 3.3 2.8 4.5 5.8 4.2 3.1 3.7 4.9 5.5 3.9 0 m 2m 3m m 0 m 2m 2m m 0 m = → Softmax The bias grows linearly with distance, but softmax normalises. Even at very long sequences (never seen in training), the bias pattern extrapolates naturally. Models using ALiBi: BLOOM (176B) MPT-7B MPT-30B OpenLLaMA*
Figure 1: ALiBi subtracts m × |i−j| from each attention score. Darker red = larger penalty for farther distance. The pattern extrapolates perfectly to any sequence length.

Why It Extrapolates

Standard PE trains on sequences of length L. Beyond L, the model has never seen those position indices and performance degrades.

ALiBi never uses position indices at all — only distances. At inference on a 4096-token sequence (when training was on 1024), the model sees distances like 1, 2, 3, … 4095 — but all it needs to do is subtract m × distance. The penalty formula works at any distance, so extrapolation is essentially free.

The paper reports that models trained at 1024 tokens with ALiBi outperform sinusoidal and learned PE baselines even at inference lengths of 2048 and 4096 tokens.

Trade-Offs

  • Pro: Zero extra parameters, trivially simple.
  • Pro: Excellent out-of-the-box extrapolation.
  • Con: The linear bias is a strong inductive bias — locality is built in. Some tasks (e.g., cross-document retrieval) may prefer more flexible attention patterns.
  • Con: Slightly outperformed by RoPE + YaRN in ultra-long-context regimes.

✅ Key Takeaways

  • ALiBi adds no PE vectors — just subtracts m × |i−j| from each attention score.
  • Different slopes m per head allow some heads to focus locally, others globally.
  • Extrapolates to longer sequences than training because it only uses relative distance, never absolute position indices.
  • Used in BLOOM (176B) and MPT; now somewhat superseded by RoPE + YaRN for long-context LLMs.

You May Also Enjoy

GraphSAGE: Inductive Learning on Large Graphs

3 minute read

Published:

GCN and GAT learn embeddings for fixed graphs — add a new node and you’re stuck. GraphSAGE (Hamilton et al., 2017) learns an aggregation function instead, so it can generate embeddings for entirely new nodes at inference time.

GAT: Graph Attention Networks

3 minute read

Published:

GCN assigns the same (degree-based) weight to every neighbour. GAT learns which neighbours actually matter — using attention coefficients on edges. More expressive, more interpretable.

GCN: Graph Convolutional Networks

3 minute read

Published:

GCN (Kipf & Welling, 2016) is the ‘hello world’ of GNNs. It simplifies spectral graph convolution into a single elegant layer: normalised neighbourhood averaging with a learned linear transformation.