Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Relative Positional Encodings: It’s All About Distance

4 minute read

Published:

TL;DR: Relative PE encodes how far apart two tokens are rather than where each one sits. Shaw et al. (2018) add learnable distance vectors to the attention computation; T5 adds a simpler learned scalar bias per distance bucket. Both approaches generalise better than absolute PE.
Paper: "Self-Attention with Relative Position Representations"  ·  arXiv:1803.02155
Authors: Peter Shaw, Jakob Uszkoreit, Ashish Vaswani
Venue: NAACL 2018  ·  📄 Read the paper
First page of the Self-Attention with Relative Position Representations paper
Paper preview — Self-Attention with Relative Position Representations (Shaw et al., 2018).
T5 and relative positional bias
T5 popularised a lightweight learned relative attention bias instead of absolute positional vectors (Raffel et al., 2020).
What relative PE gets right: for many language tasks, distance matters more than absolute index. "one token back" is often reusable; "position 387" usually is not.

The Problem with Absolute Position

Absolute PEs assign a position vector based on where a token sits in the sequence — position 0, position 1, etc.

But think about what really matters for attention: whether token A is close to or far from token B, not exactly where either one sits in a global index. The word “love” in “I love dogs” and “I really truly love dogs” has the same relationship to “dogs” (it’s the verb, directly preceding the object) — even though its absolute position is different.

Relative PE captures exactly this intuition.

Shaw et al. (2018): Relative Attention

Shaw, Uszkoreit, and Vaswani modify the attention score between token i and token j to include a learned relative position embedding a_{ij}:

score(i, j) = (q_i · k_j + q_i · a_{ij}) / √d_k

Here a_{ij} is the embedding for the clipped relative distance clip(i − j, −k, k). A maximum distance k (e.g., 16) is used — beyond that, all distances share the same embedding.

Relative Distance Between Token Pairs I love dogs and cats anchor: "dogs" dist = −2 dist = −1 dist = +1 dist = +2 Key insight: same relative distances in any sentence position "love" → "dogs" is always distance +1, whether at position 3, 7, or 25 globally. Shaw et al. (2018) Learned vector a_{ij} added to QK attention Also modifies V computation T5 Relative Bias (2020) Learned scalar bias b_{ij} added to attention logits Simpler, buckets for distance, shared across layers
Figure 1: "dogs" attending to tokens at relative distances −2, −1, +1, +2. The same distance embeddings apply regardless of absolute position.

T5 Relative Bias

Raffel et al. (T5, 2020) simplify further. Instead of a full vector per relative position, they add a learned scalar bias to the attention score:

score(i, j) = q_i · k_j / √d_k + b(i − j)

b(·) is a small lookup table of scalars, indexed by bucketed distances. Nearby distances (−1, 0, +1) each get their own bucket; farther distances share buckets. The biases are shared across all layers but learned separately per attention head.

This is extremely memory-efficient and generalises gracefully to longer sequences.

Concrete Worked Example

Suppose we have the sentence: “I love dogs and cats” (5 tokens).

With T5-style relative bias and m (head slope) = 0.5, the attention score from token 2 (“love”) to each other token gets a bias added:

Attending from “love” (pos 1)target posdistancebias b(dist)
“I”0−1b(−1)
“love”10b(0) = 0
“dogs”2+1b(+1)
“and”3+2b(+2)
“cats”4+3b(+3)

The biases b(−1), b(0), b(+1), … are small learned scalars. Crucially, if this same sentence fragment appeared at positions 50–54 in a longer document, the exact same bias values apply — because only the relative distances matter, not the absolute indices.

Why this matters at scale: a model trained on sequences of 512 tokens using relative PE can still correctly apply the bias b(+1) at positions 1000 and 1001 — it has seen "distance 1" millions of times. An absolute-PE model at position 1000 has seen that index far less often.

Why Relative PE Generalises Better

Absolute PE puts a token at “position 42” — if training sequences were at most 64 long, the model learned what position 42 means. At position 200? It never saw that index.

Relative PE never mentions global positions. It only says “these two tokens are 5 apart.” As long as the model has seen pairs 5 apart before (which it almost certainly has), it can handle any sequence length.

✅ Key Takeaways

  • Relative PE encodes the gap between pairs of tokens, not their absolute position.
  • Shaw et al. (2018) adds a learned vector to the QK dot-product; T5 adds a simpler scalar bias.
  • Generalises better to longer sequences — the model never sees an "unseen absolute position".
  • T5-style relative biases are lightweight (one small table) and used across Flan-T5, Switch Transformer, and more.

You May Also Enjoy

Modern Activation Functions: GELU, SiLU, Mish, and Smooth Gating

11 minute read

Published:

Once ReLU became the default, researchers started asking a better question: can we keep the easy optimization while making the activation smoother, softer, and more expressive? This chapter covers the modern answers.

Activation Functions in Neural Networks: Why Non-Linearity Matters

11 minute read

Published:

Activation functions are the reason neural networks can model curved decision boundaries instead of collapsing into one giant linear map. This chapter builds the intuition first, then walks through the classical functions that shaped deep learning.

FoPE: Fourier Position Embedding for Length Generalization

6 minute read

Published:

FoPE rethinks long-context positional encoding from a frequency-domain perspective. Instead of only stretching RoPE heuristically, it explicitly improves attention’s periodic extension so Transformers generalize more gracefully to longer sequences.