Vision State Space Models
Vision State Space Models (VSSM) are the sequence modeling successors that treat images as flattened sequences and apply linear state-space recurrences to achieve global receptive fields with linear time — by combining state-space layers (such as S4) with convolutional input/output projections, VSSMs process very long vision sequences without the quadratic bottleneck of attention.
What Is a Vision State Space Model?
- Definition: An architecture that views each image as a 1D token stream and feeds it through state-space layers that update an internal hidden state using linear recurrences, followed by output projections that reshape the state into patches.
- Key Feature 1: SSM layers maintain global context through linear time updates, so they do not require sparse or windowed attention.
- Key Feature 2: Input/output convolutions map between 2D patches and the 1D sequence expected by the SSM layer.
- Key Feature 3: Parameterized kernels (e.g., hi-parameterization or power series) control the memory of the recurrence.
- Key Feature 4: VSSMs often surround the state-space block with residual connections and normalization to match transformer-style training.
Why VSSM Matters
- Linear Complexity: Compute grows linearly with sequence length, enabling video or gigapixel images to be processed affordably.
- Global Context: The recurrence inherently mixes all tokens, so even long-range dependencies are captured without explicit attention patterns.
- Robustness: Deterministic recurrences can be more stable than stochastic attention, especially on streaming inputs.
- Hardware Friendliness: State-space layers use matrix-vector products similar to convolutions, making them easy to optimize on chips.
- Complementary: VSSMs can replace only the attention blocks in a hybrid transformer, keeping other components unchanged.
State Space Choices
S4 (Structured State Space):
- Uses parameterized kernels derived from HiPPO matrices for long memory.
- Offers exponential decay that matches both short and long contexts.
Liquid S4:
- Adds gating mechanisms to mix multiple SSMs.
- Improves expressivity with minimal compute overhead.
Kernelized Recurrences:
- Use learned kernels that define the impulse response rather than fixed matrices.
- Provide fine control over temporal decay.
How It Works / Technical Details
Step 1: Flatten the image into a 1D sequence of patch embeddings, feed it through a convolutional projection to match the SSM input dimension, and pass it through the state-space recurrence which updates a hidden state per step.
Step 2: Project the resulting sequence back to tokens, add residual connections, and reshape into spatial patches for downstream layers or heads.
Comparison / Alternatives
| Aspect | VSSM | Linear Attention | Standard ViT |
|---|---|---|---|
| Complexity | O(N) | O(N) | O(N^2) |
| Global Context | Yes | Yes | Yes |
| Streaming | Excellent | Excellent | Limited |
| Implementation | More novel | Medium | Standard |
Tools & Platforms
- StateSpaceModels repo: Contains S4 and Liquid S4 implementations for vision tasks.
- FlashAttention: Can fuse state-space recurrences with minimal overhead during inference.
- Hugging Face: Some models include state-space encoders as alternatives to attention.
- Profilers: Monitor token throughput to confirm linear scaling gains.
Vision SSMs are the recurrence-based alternative to attention that keeps the entire token stream within reach while staying linear in length — they bring the robustness of signal processing to modern vision architectures.