Fnet
FNet is a Transformer alternative that replaces the self-attention sublayer entirely with a parameter-free Fourier Transform, demonstrating that simple token mixing via the Fast Fourier Transform (FFT) can achieve 92-97% of BERT's accuracy on standard NLP benchmarks while training 80% faster on GPU and 70% faster on TPU. FNet shows that much of a Transformer's power comes from the feed-forward layers, not the attention mechanism.
Why FNet Matters in AI/ML:
FNet challenged the assumption that attention is essential for Transformer performance, demonstrating that a simple, fixed linear transform (FFT) provides sufficient token mixing for most NLP tasks, raising fundamental questions about what makes Transformers effective.
• Fourier sublayer — Each Transformer layer replaces multi-head self-attention with a 2D DFT: first along the sequence dimension (mixing tokens) and then along the hidden dimension (mixing features); this is computed using the FFT in O(N log N) time with zero learnable parameters
• No attention parameters — FNet eliminates all Q, K, V projection matrices, attention heads, and output projections; the Fourier transform provides global token mixing through frequency-domain decomposition with no trainable weights in the mixing layer
• Feed-forward dominance — FNet's competitive performance reveals that the feed-forward network (FFN) sublayers—not attention—are responsible for most of the Transformer's representational power; attention primarily provides input-dependent token mixing that the FFT approximates
• Training speed — Without attention computation (which is memory-bound on GPUs), FNet achieves 7× faster training throughput on GPU for long sequences and 2× faster for short sequences compared to standard BERT
• Hybrid architectures — Replacing only some attention layers with Fourier layers (e.g., attention in the first 2 layers, FFT in the rest) recovers 99%+ of BERT performance while maintaining most of FNet's speed advantage
| Property | FNet | BERT | Hybrid (2 attn + FFT) |
|---|---|---|---|
| Token Mixing | FFT (fixed) | Attention (learned) | Both |
| Mixing Parameters | 0 | O(d²·heads) per layer | Reduced |
| GLUE Score | ~92% of BERT | Baseline | ~99% of BERT |
| Training Speed (GPU) | 7× faster (long seq) | 1× | 2-3× faster |
| Sequence Complexity | O(N log N) | O(N²) | Mixed |
| Input Dependence | None (fixed mixing) | Full (data-dependent) | Partial |
FNet is a landmark study demonstrating that parameter-free Fourier transforms can replace learned attention mechanisms with minimal accuracy loss, fundamentally challenging the centrality of attention in Transformer architectures and revealing that feed-forward layers—not attention—are the primary source of representational power in modern language models.