Learned Position Embeddings

Learned position embeddings are trainable parameter vectors assigned to each spatial position in a Vision Transformer's input sequence — providing the model with spatial location information by adding a unique, learned vector to each patch token so the transformer can distinguish where in the image each patch originated.

What Are Learned Position Embeddings?

• Definition: A set of trainable vectors, one per input sequence position, that are added to the patch embeddings before processing by transformer encoder layers. For ViT-Base with 196 patches + 1 CLS token, this is a learnable parameter matrix of shape (197, 768).
• Origin: Derived from the original Transformer architecture (Vaswani et al., 2017) and adapted for vision by ViT (Dosovitskiy et al., 2020).
• Initialization: Typically initialized randomly (normal or uniform distribution) and optimized during training through backpropagation like any other model parameter.
• Addition Operation: Position information is injected by element-wise addition: token_input = patch_embedding + position_embedding[i] for position i.

Why Learned Position Embeddings Matter

• Spatial Awareness: Without position embeddings, the transformer treats the input as a bag of patches with no spatial ordering — it cannot distinguish top-left from bottom-right, making spatial reasoning impossible.
• Permutation Invariance Problem: Self-attention is inherently permutation-equivariant — the output is the same regardless of input ordering. Position embeddings break this symmetry and inject spatial structure.
• Simplicity: Learned embeddings are the simplest position encoding — just add a parameter matrix. No special implementation, no mathematical formulas, no architectural modifications.
• Task Adaptation: The model can learn task-specific position patterns — for classification, it might learn center-weighted position biases; for detection, it might learn edge-aware position patterns.
• Empirical Baseline: Learned position embeddings remain a strong baseline — the original ViT showed minimal difference between learned and fixed sinusoidal position embeddings.

How Learned Position Embeddings Work

Training Phase:
- Initialize position_embedding as a learnable nn.Parameter of shape (N+1, D).
- At each forward pass: x = patch_embed(image) + position_embedding.
- Gradients flow through position embeddings during backpropagation.
- The model learns to assign vectors that encode useful spatial information.

What the Model Learns:
- Analysis of trained position embeddings reveals clear spatial structure:
- Nearby positions have similar embeddings (high cosine similarity).
- Same-row and same-column positions show strong correlation patterns.
- The 2D spatial grid structure emerges naturally despite being stored as a 1D list.
- Corner and edge positions are distinct from center positions.

Limitations of Learned Position Embeddings

Resolution Transfer Protocol

When fine-tuning a ViT at a different resolution than pretraining:
1. Reshape 1D position embeddings to 2D grid: (N,) → (H_train, W_train).
2. Apply bicubic interpolation to new grid: (H_train, W_train) → (H_new, W_new).
3. Flatten back to 1D: (H_new × W_new,).
4. Fine-tune with the interpolated position embeddings (typically with lower learning rate for positions).

Learned Position Embeddings vs. Alternatives

Learned position embeddings are the simplest and most intuitive spatial encoding for Vision Transformers — while newer alternatives offer better resolution flexibility and translation invariance, learned embeddings remain the default choice in many architectures due to their simplicity, strong baseline performance, and ease of implementation.