Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

RoPE: Rotary Position Embeddings

3 minute read

Published:

TL;DR: RoPE (Su et al., 2021) applies a position-dependent rotation matrix to query and key vectors. The key mathematical property: the dot product q·k only depends on the relative distance between positions — giving you relative attention from an absolute encoding. No extra parameters.

The Motivation

The ideal PE would:

  1. Inject absolute position information (so the model knows where each token is).
  2. Produce attention scores that depend only on relative distances (so the model generalises to longer sequences).
  3. Require no extra parameters.

Absolute PE achieves (1) but not (2). Relative PE achieves (2) but adds complexity. RoPE achieves all three simultaneously.

The Rotation Intuition

Think of each 2D pair of embedding dimensions as a 2D vector. Rotating it by an angle θ×position is like rotating a clock hand: the absolute angle encodes position, but two hands’ relative angle encodes their difference.

RoPE applies this idea to the entire d-dimensional embedding by treating it as d/2 pairs of 2D coordinates, each rotated by a different frequency.

Query q at pos = 1 original q R(θ·1)·q θ×1 Key k at pos = 3 original k R(θ·3)·k θ×3 Key Property R(θ·pos_q)·q · R(θ·pos_k)·k = q · R(θ·(pos_k − pos_q))·k Dot product only depends on the relative distance pos_k − pos_q !
Figure 1: RoPE rotates q by angle θ×pos_q and k by θ×pos_k. Because rotation is linear, the dot product between rotated vectors encodes only the relative distance pos_k − pos_q.

The Math (Simplified)

For a 2D pair of embedding dimensions (x, y) at position pos:

RoPE(pos) · [x, y] = [x·cos(θ·pos) − y·sin(θ·pos), x·sin(θ·pos) + y·cos(θ·pos)]

This is just a standard 2D rotation matrix applied to (x, y) with angle θ × pos. For the full d-dimensional embedding, d/2 independent rotation matrices are applied — each pair at a different frequency (like sinusoidal PE).

The crucial identity: (R(m)·q) · (R(n)·k) = q · (R(n−m)·k) — the dot product depends only on n − m, the relative distance.

Why LLMs Love RoPE

Models using RoPE:

LLaMA 2 & 3 Mistral Mixtral GPT-NeoX Falcon Gemma Qwen Yi

RoPE dominates modern LLM training because:

  1. No extra parameters — the rotation is computed on the fly.
  2. Relative attention from absolute encoding — the best of both worlds.
  3. Good extrapolation — with extensions like YaRN (Yet another RoPE extensioN), models trained on 4K tokens can serve 128K.
  4. Compatible with KV caching — rotations can be precomputed and cached efficiently.

RoPE Extensions for Long Context

Standard RoPE degrades when pushed far beyond training length. Several extensions fix this:

  • Position Interpolation: scale positions down to fit training range.
  • YaRN: different scaling for different frequency groups; currently the most popular approach.
  • LongRoPE / LongLLaMA: progressive context extension during fine-tuning.

✅ Key Takeaways

  • RoPE rotates Q and K vectors by an angle proportional to position — no extra parameters.
  • The dot product of rotated Q and K depends only on relative position, not absolute — the best of both worlds.
  • Used in virtually every top-performing open-weight LLM: LLaMA 3, Mistral, Gemma.
  • Extensions like YaRN enable far longer contexts than the training length.

You May Also Enjoy

GraphSAGE: Inductive Learning on Large Graphs

3 minute read

Published:

GCN and GAT learn embeddings for fixed graphs — add a new node and you’re stuck. GraphSAGE (Hamilton et al., 2017) learns an aggregation function instead, so it can generate embeddings for entirely new nodes at inference time.

GAT: Graph Attention Networks

3 minute read

Published:

GCN assigns the same (degree-based) weight to every neighbour. GAT learns which neighbours actually matter — using attention coefficients on edges. More expressive, more interpretable.

GCN: Graph Convolutional Networks

3 minute read

Published:

GCN (Kipf & Welling, 2016) is the ‘hello world’ of GNNs. It simplifies spectral graph convolution into a single elegant layer: normalised neighbourhood averaging with a learned linear transformation.