Dense Model

Dense models activate all parameters for every input, the standard architecture for most neural networks. Definition: Every parameter participates in every forward pass. All weights used for all inputs. Contrast with sparse: Sparse/MoE models activate only subset of parameters per input. Computation: For dense transformer, FLOPs scale directly with parameter count. Larger model = more compute per token. Memory: All parameters must be in memory for inference. 70B model needs significant GPU memory. Training: Straightforward optimization. All parameters receive gradients every step. Advantages: Simpler architecture, well-understood training dynamics, consistent behavior across inputs. Disadvantages: Compute scales linearly with params. Eventually compute-inefficient at extreme scale. Examples: GPT-4 (rumored partially MoE but mostly dense), LLaMA, Claude, most deployed LLMs. Trade-off with sparse: Dense models have better predictable behavior; sparse models can be larger for same compute. Current practice: Dense remains dominant for most production deployments due to simplicity and reliability.