Perceiver
Perceiver is a general-purpose transformer architecture that uses cross-attention to project arbitrary-size inputs into a fixed-size latent array — decoupling the computational cost from input size so that a 100K-pixel image, a 50K-token audio clip, and a 10K-point cloud all get processed through the same small latent bottleneck (e.g., 512 latent vectors), enabling a single architecture to handle any modality without modality-specific design choices.
What Is Perceiver?
- Definition: A transformer architecture (Jaegle et al., 2021, DeepMind) where the input (of any size) is processed through cross-attention with a small learned latent array (typically 256-1024 vectors), and all subsequent self-attention operates on this compact latent space rather than the high-dimensional input space.
- The Problem: Standard transformers apply O(n²) self-attention directly on the input. For a 224×224 image (50K pixels), that's 2.5 billion attention computations per layer — impossible. CNNs and ViTs work around this with patches, but each modality needs custom architecture.
- The Solution: Project ANY input into a fixed-size latent array via cross-attention (cost: O(n × M) where M is latent size << n), then apply self-attention only on the small latent array (cost: O(M²), independent of input size).
Architecture
| Step | Operation | Input | Output | Complexity |
|---|---|---|---|---|
| 1. Cross-Attention | Latent queries attend to input | Latent: M × d, Input: N × d_in | M × d (latent updated) | O(M × N) |
| 2. Self-Attention | Latent self-attention (multiple blocks) | M × d | M × d (refined) | O(M²) per block |
| 3. Repeat (optional) | Additional cross-attention + self-attention | Updated latent + original input | M × d (further refined) | O(M × N + M²) |
| 4. Decode | Task-specific output (class token, etc.) | M × d | Task output | O(M) |
Key Insight: The Latent Bottleneck
| Property | Standard Transformer | Perceiver |
|---|---|---|
| Self-attention cost | O(N²) — depends on input size | O(M²) — depends on latent size (fixed) |
| Input flexibility | Fixed tokenization per modality | Any byte array, any modality |
| Scalability | Cost grows quadratically with input | Cost fixed regardless of input size |
| Architecture per modality | Different: ViT for images, BERT for text | Same architecture for everything |
Example: M=512 latents, N=50,000 input elements:
- Standard: Self-attention = 50,000² = 2.5B operations per layer
- Perceiver: Cross-attn = 512 × 50,000 = 25.6M; Self-attn = 512² = 262K per block
Modality Flexibility
| Modality | Input Representation | Same Perceiver Architecture |
|---|---|---|
| Images | Pixel array (H×W×C) with positional encoding | ✓ |
| Audio | Raw waveform or spectrogram | ✓ |
| Point Clouds | 3D coordinates (N×3) | ✓ |
| Video | Pixel frames (T×H×W×C) | ✓ |
| Text | Token embeddings | ✓ |
| Multimodal | Concatenate all modalities as one input array | ✓ |
Perceiver is the universal perception architecture — using cross-attention to a fixed-size latent array to decouple computational cost from input size and modality, enabling a single unchanged architecture to process images, audio, video, point clouds, and multimodal inputs with O(M²) self-attention cost regardless of whether the input has 1,000 or 1,000,000 elements, pioneering the movement toward truly modality-agnostic deep learning.