Position Encoding Interpolation

Position encoding interpolation is the method for resizing learned positional embeddings when ViT input resolution changes and token grid dimensions no longer match - by interpolating positional maps from old grid to new grid, pretrained knowledge can transfer to larger or smaller resolutions without reinitializing the model.

What Is Position Encoding Interpolation?

• Definition: Numerical resizing of 2D positional embedding grid, often using bicubic interpolation, to fit a new patch layout.
• Need: Pretrained positional table for 14x14 grid cannot directly map to 24x24 grid.
• Common Method: Separate class token embedding, interpolate only spatial tokens, then concatenate back.
• Goal: Preserve relative spatial priors learned during pretraining.

Why It Matters

• Checkpoint Reuse: Enables smooth transfer from low resolution pretraining to high resolution fine-tuning.
• Stability: Avoids random reinitialization of positional parameters.
• Performance Retention: Maintains strong baseline accuracy after resolution change.
• Implementation Simplicity: One preprocessing step with significant practical impact.
• Versatility: Works for classification, detection, and segmentation backbones.

Interpolation Options

Bicubic Interpolation:
- Most common due to smooth and stable resizing.
- Good balance of quality and speed.

Bilinear Interpolation:
- Faster and simpler but slightly less smooth.
- Acceptable in some production pipelines.

Learned Reprojection:
- Train small adapter to map old positional table to new shape.
- Can outperform fixed interpolation when large shifts occur.

How It Works

Step 1: Extract class token position embedding and reshape spatial embeddings to 2D grid from original checkpoint.

Step 2: Interpolate spatial grid to target size, flatten back to sequence, and concatenate class token embedding.

Tools & Platforms

• timm: Built in utility functions for positional interpolation.
• Hugging Face ViT: Includes checkpoint adaptation helpers.
• Custom loaders: Easy to integrate into fine-tuning entry points.

Position encoding interpolation is the key compatibility bridge that allows ViT checkpoints to move across resolutions without losing learned spatial priors - it is a required step in nearly every high resolution transfer workflow.