Learnable Position Embedding

Learnable Position Embedding is a position encoding method where position vectors are treated as trainable parameters — each position in the sequence has its own learned embedding vector that is added to the token embedding, allowing the model to discover optimal position representations.

How Does It Work?

• Parameters: $P \in \mathbb{R}^{N_{max} \times d}$ — one $d$-dimensional vector per position.
• Application: $x_i' = x_i + P_i$ (add position embedding to token embedding).
• Training: Position vectors are optimized via backpropagation alongside all other parameters.
• Used In: BERT, GPT-2, ViT, most modern transformers.

Why It Matters

• Simplicity: The simplest position encoding — just add learned vectors.
• Flexibility: The model discovers whatever positional patterns are useful for the task.
• Limitation: Fixed maximum sequence length. Cannot generalize to longer sequences than training.

Learnable Position Embedding is the model teaching itself about position — letting optimization discover the best way to encode sequential or spatial position.