Large Language Models

# Large Language Models: The Memory Wall, the Roofline, and Hardware Choice

Inside Decoder-Only Self-Attention — "Attention Is All You Need" every generated token looks back at every prior token through one equation: softmax(QKᵀ/√d)V one decoder block, causal (masked) self-attention "The" "cat" "sat" "on" (query) token embeddings + positions Q = x·W_Q K = x·W_K V = x·W_V scores = Q·Kᵀ / √d causal mask "on" can't see future tokens softmax(scores) → attention weights "The" "cat" "sat" "sat" gets the highest weight here — the direct subject-verb context output = weights · V → context vector for "on" this one block repeats across every layer and every head — multi-head attention runs several Q/K/V projections in parallel, each learning a different kind of relationship between tokens

A large language model (LLM) is, from a hardware perspective, an enormous sequence of matrix multiplications with an attention mechanism threaded through it. What makes LLMs a distinct *hardware* problem — not just a bigger version of prior deep learning workloads — is the ratio between how much data has to move and how much arithmetic gets done with it. That ratio, called arithmetic intensity, determines whether a given LLM workload runs at a chip's peak compute speed or is stuck waiting on memory, and it differs sharply between training and inference.

---

## 1. The Roofline Model and the Memory Wall

Every chip has two hard ceilings: a peak compute rate (FLOPs/s) and a peak memory bandwidth (bytes/s). For any given operation, arithmetic intensity = FLOPs performed ÷ bytes moved. The roofline model plots achievable throughput against arithmetic intensity and shows exactly where a workload sits relative to those two ceilings.

Roofline Model — Where LLM Phases Sit log-log axes: achievable throughput vs. arithmetic intensity arithmetic intensity (FLOPs/byte) → throughput (FLOPs/s) → memory bandwidth roof (slope = BW) compute roof (peak Tensor Core FLOPs/s) ridge point (I_ridge ≈ 150 FLOP/B) Memory-Bound Compute-Bound Decode (GEMV) ~1 FLOP/byte Small-Batch Prefill Large-Batch Training (GEMM) >150 FLOP/byte Any point left of the ridge is capped by the memory bandwidth roof — faster ALUs provide zero speedup.

Autoregressive decoding — generating one token at a time, re-reading model weights and the KV cache on every step — has very low arithmetic intensity: a small amount of compute per byte moved. It sits firmly on the memory-bound slope, meaning faster compute units bring almost no speedup; only more memory bandwidth, smaller model footprint (quantization), or bigger batches (amortizing the same weight read over more tokens) move the needle.

---

## 2. Attention Matrix Multiplication Cost

The two matrix multiplications inside attention — $Q \cdot K^T$ and $\text{softmax}(\cdot) \cdot V$ — both scale as $\mathcal{O}(\text{sequence\_length}^2 \times \text{head\_dim})$, while the feed-forward layers scale as $\mathcal{O}(\text{sequence\_length} \times \text{hidden\_dim}^2)$. At short sequence lengths the feed-forward cost dominates; past a crossover length (roughly where $\text{seq\_len} \approx \text{hidden\_dim}$), the quadratic attention term takes over and long-context inference becomes dominated by attention rather than the dense layers. This is the direct hardware motivation for fused attention kernels and for architectural variants (grouped-query attention, sliding-window attention) that cut the quadratic term.

---

## 3. Inference vs. Training: The Autoregressive Memory Wall Dynamics

Autoregressive LLM Execution — The Memory Wall Dynamic prefill runs at compute capacity — decode collapses into memory bandwidth starvation Generated Token Sequence Length (Tokens) → Arithmetic Intensity (FLOPs/Byte) → KV Cache Footprint (GB) → Prefill Phase >150 FLOP/B (GEMM) Autoregressive Decode Collapse (~1 FLOP/Byte · GEMV) KV Cache Linear Growth M_kv = 2 · b · s · L · d_head · n_kv HBM Capacity Ceiling (OOM Boundary) Decoding 1 token requires streaming all weights + KV cache from HBM — arithmetic intensity < 1 FLOP/byte. This is why LLM inference is memory-bandwidth bounded, requiring GQA, FP8/FP4 quantization, and continuous batching.

Training optimizes for sustained throughput across a large, synchronized multi-chip cluster, where arithmetic intensity is pushed high by large batches and the dominant hardware constraint is interconnect bandwidth for gradient synchronization. Inference optimizes for latency and cost per token on far smaller hardware footprints, where decode's low arithmetic intensity makes memory bandwidth and KV-cache size — not peak FLOPs — the binding constraint. The same model, run in these two modes, wants genuinely different hardware.

---

## 4. Key Takeaway

Large language model performance is governed by arithmetic intensity, not FLOPs on a spec sheet: the roofline model shows that decode sits on the memory-bound slope where more compute buys nothing, attention's quadratic cost in sequence length determines when long-context workloads stop being feed-forward-dominated, and training versus inference demand genuinely different hardware shapes — high-bandwidth multi-chip fabrics for training's gradient synchronization versus low-latency, quantization-friendly single-chip paths for inference's memory-bound decode loop. Understanding which side of the memory wall a given LLM workload sits on is the single most useful fact for choosing or designing hardware for it.

---

## References

  • Williams, Waterman & Patterson, "Roofline: An Insightful Visual Performance Model for Multicore Architectures," CACM 2009.
  • Kaplan et al., "Scaling Laws for Neural Language Models" — compute and data scaling behavior.
  • Pope et al., "Efficiently Scaling Transformer Inference," MLSys 2023.
  • Rajbhandari et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models," SC 2020 — training memory footprint breakdown.

Take Large language models further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app