Feedforward

The feedforward network (FFN/MLP) in transformers processes each position independently after attention, typically expanding to 4× hidden dimension then projecting back, containing the majority of the model's parameters and computational cost. FFN structure: two linear projections with nonlinearity: FFN(x) = W_2 × ReLU(W_1 × x + b_1) + b_2, where W_1 projects to 4× dimension and W_2 projects back. Parameter distribution: for d=1024, W_1 is 1024×4096, W_2 is 4096×1024—8M parameters per layer versus ~3M for attention. This means ~70% of transformer parameters are in FFNs. Computational role: FFNs process each position with the same transformation (position-wise), providing: nonlinear transformation (attention is mostly linear), capacity/memorization (key-value memory interpretation), and feature mixing (combining attention outputs). Activation functions: GELU replaced ReLU in modern models (smoother, better performance), SwiGLU/GeGLU provide gated activation with improved quality. FFN as memory: recent interpretations suggest FFN weights store factual knowledge, with first layer as key lookup and second as value retrieval. Optimization: FFNs are embarrassingly parallel across positions, dominate training FLOPs, and are primary targets for sparsity (Mixture of Experts) and quantization.