Sinusoidal Positional Encodings: The Original Solution

Intuition First: Why Waves?

Imagine you want to give each position in a sequence a unique “fingerprint” using only values between −1 and +1. A single sine wave won’t work — it repeats. But if you stack many sine waves at different frequencies, their combined values at any position form a unique signature (like a barcode).

That’s exactly what sinusoidal PE does: each dimension of the encoding vector is one wave of a specific frequency. Low-index dimensions oscillate fast; high-index dimensions oscillate slowly. Together, they uniquely identify any position.

The Formula

For a token at position pos, dimension i of its PE vector is:

That’s it. Even dimensions get a sine, odd dimensions get a cosine. The frequency shrinks geometrically as i increases.

The Intuition: A Continuous Binary Counter

Think of binary numbers: 0001, 0010, 0011, 0100, … The rightmost bit flips every step (high frequency); the leftmost bit flips rarely (low frequency). Together they uniquely identify each integer.

Sinusoidal PE does the same in a continuous, smooth way:

• High dimensions (i small → high frequency): the sin/cos oscillates rapidly, capturing fine-grained position differences.
• Low dimensions (i large → low frequency): the sin/cos changes slowly, encoding coarse position.

Each position gets a unique fingerprint — a mix of fast and slow oscillations — that the model can read.

Concrete Worked Example (d = 4)

Let’s compute the PE vector for position pos = 1 with d = 4 dimensions (i = 0 and i = 1):

PE(1, dim=0) = sin(1 / 10000^(0/4)) = sin(1 / 1)       = sin(1.0)  ≈  0.841
PE(1, dim=1) = cos(1 / 10000^(0/4)) = cos(1 / 1)       = cos(1.0)  ≈  0.540
PE(1, dim=2) = sin(1 / 10000^(2/4)) = sin(1 / 100)     = sin(0.01) ≈  0.010
PE(1, dim=3) = cos(1 / 10000^(2/4)) = cos(1 / 100)     = cos(0.01) ≈  1.000

So PE(pos=1) ≈ [0.841, 0.540, 0.010, 1.000].

Now compare pos = 2:

PE(2, dim=0) = sin(2.0) ≈  0.909    (changed a lot — high frequency)
PE(2, dim=2) = sin(0.02) ≈ 0.020    (barely changed — low frequency)

The high-frequency dims (left) distinguish nearby positions; the low-frequency dims (right) distinguish distant ones. Together they uniquely encode every position.

Three Key Properties

1. Uniqueness. The combination of many frequencies produces a unique vector for each position — like a fingerprint. Two positions will never have the same PE vector.

2. Smooth transitions. Adjacent positions have similar PE vectors. The model can learn that nearby positions are related without any explicit guidance.

3. Relative encoding via dot products. The dot product PE(pos₁) · PE(pos₂) depends only on the distance pos₁ − pos₂. This means the model can implicitly reason about relative distances from absolute positions — a crucial and non-obvious property.

Why Use 10000?

The base 10000 is chosen so that the wavelengths span from 2π (highest frequency, dim 0) to 10000·2π (lowest frequency, last dim). This gives the model coverage over positions from 1 to roughly 10,000 tokens — sufficient for most early use cases.

Limitations

• Fixed formula, so it can’t be fine-tuned for a specific task.
• Extrapolation beyond the training length is imperfect, though better than learned absolute PEs.
• Modern LLMs (with 128K+ context windows) need better solutions — enter RoPE and ALiBi.