Sinusoidal Position Encoding

Sinusoidal Position Encoding is the original position encoding from the Transformer paper — using fixed sine and cosine functions at different frequencies to encode absolute position, based on the idea that relative positions can be represented as linear transformations.

How Does It Work?

• Formula: $PE_{(pos, 2i)} = sin(pos / 10000^{2i/d})$, $PE_{(pos, 2i+1)} = cos(pos / 10000^{2i/d})$
• Frequencies: Each dimension uses a different frequency, from high (position 0) to low (position $d-1$).
• Relative Position: $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$ for any fixed $k$.
• Paper: Vaswani et al. (2017).

Why It Matters

• No Parameters: Completely deterministic — no learnable parameters for position encoding.
• Extrapolation: Can theoretically encode positions beyond the training length.
• Foundation: Inspired RoPE, ALiBi, and other modern position encodings.

Sinusoidal Position Encoding is the mathematical clock of the original Transformer — encoding position through harmonic frequencies at no parameter cost.