Conditional Position Encoding

Conditional Position Encoding (CPE) is a position encoding method that generates position-dependent features conditioned on the input content — using a depth-wise convolution on the input tokens to implicitly encode positional information.

How Does CPE Work?

• Mechanism: Apply a depth-wise convolution (e.g., 3×3) to the token sequence/grid.
• Implicit Position: The convolution with zero-padding naturally encodes boundary positions. Interior positions are derived from local neighborhood structure.
• Dynamic: The positional information adapts to the input content, unlike fixed sinusoidal encodings.
• Paper: Chu et al. (2021, Conditional Positional Encodings for ViT).

Why It Matters

• No Explicit PE: Eliminates the need for explicit position embeddings (learnable or sinusoidal).
• Resolution Flexible: Works at any resolution without interpolation (CNNs are inherently resolution-agnostic).
• Integration: Can be added at any layer, not just the input, providing positional information throughout the network.

CPE is implicit position from local structure — using convolutions to derive positional information from the spatial arrangement of features.