Transformer Chips

# Transformer Chips: Attention Engines and KV Cache Bandwidth

Transformer chips are AI accelerators whose datapath and memory system are shaped specifically around the attention operation and the autoregressive decode loop, rather than convolution or generic dense matmul. Attention introduces a data-dependent, quadratic-in-sequence-length access pattern that ordinary tensor cores handle inefficiently — transformer-optimized silicon adds dedicated attention engines, a KV-cache-aware memory path, and low-precision (FP8/FP4) execution to close that gap.

---

## 1. Why Attention Needs Its Own Hardware Path

A transformer layer computes `softmax(QKᵗ / √d) · V` for every token against every other token it attends to. Unlike a weight matrix in a dense layer, the key/value (KV) tensors are generated per sequence, at runtime, and grow with every new token during autoregressive decoding. That means:

  • the "weight" the engine multiplies against is not stationary — it changes every request and every step, so weight-stationary systolic dataflow (ideal for dense layers) is the wrong shape for attention;
  • the softmax normalization sits in the middle of the matmul chain, forcing a reduction step that a pure MAC array cannot do on its own;
  • the KV cache that holds past keys/values for every token grows linearly with sequence length and batch size, and must be read in full for every new token generated — making attention memory-bandwidth-bound long before it becomes compute-bound.

## 2. The Attention Engine Dataflow

Attention Engine — QKV to Weighted Output fused pipeline keeps the softmax reduction on-chip instead of round-tripping to memory Q projection current token(s) K cache all prior tokens V cache all prior tokens Score matmul: Q · Kᵗ / √d one score row per query token, one column per cached key Softmax reduction row-wise max-subtract, exponentiate, normalize — kept on-chip (flash-attention style) Weighted sum: softmax(scores) · V output token embedding for this layer fusing score matmul → softmax → weighted sum into one pass avoids writing the full attention score matrix (O(n²) in sequence length) back to memory — this fusion is the core trick in flash-attention-class kernels

Fusing the three stages — score matmul, softmax, weighted sum — into a single pass is the dominant optimization in modern transformer hardware and kernels (the flash-attention family). Without fusion, the intermediate attention score matrix, which grows quadratically with sequence length, would have to be written to and read back from memory, dwarfing the cost of the arithmetic itself.

---

## 3. KV Cache Bandwidth and Quantization

Every generated token must read the *entire* KV cache accumulated so far, every single decode step. Cache size is `2 × layers × heads × head_dim × seq_len × batch × bytes_per_element` — it scales linearly in every one of those factors, which is why long-context, large-batch inference becomes a memory bandwidth problem, not a compute problem.

KV Cache Footprint vs. Precision same sequence length and batch size, four numeric formats FP16 100% FP8 50% INT4 / FP4 25% 4-bit + sparsity ~10% halving bytes per KV element roughly halves both memory footprint and the bandwidth needed to read the cache on every decode step — the main lever for raising batch size and context length at fixed memory bandwidth cost: lower precision narrows the dynamic range available for attention scores and cached values, so FP4/INT4 caches typically need per-channel scaling or mixed precision to hold quality

FP8 KV caches are now standard in production inference stacks; FP4/INT4 caches, combined with structured sparsity or grouped quantization, push further but require careful calibration to avoid accuracy loss on long contexts where quantization error accumulates across many attention steps.

---

## 4. Prefill vs. Decode: Two Different Bottlenecks

Transformer inference has two phases with opposite hardware demands:

  • Prefill processes the entire input prompt in one pass. It is compute-bound and matmul-dominated — large batched matrix multiplies that map well onto dense tensor cores.
  • Decode generates one token at a time, re-reading the full KV cache each step. It is memory-bandwidth-bound and latency-sensitive — the systolic MAC array sits mostly idle while the chip waits on cache reads.

Transformer-optimized chips size on-chip SRAM, HBM channels, and batching strategy (continuous batching, chunked prefill) specifically to keep both phases close to their respective roofline limits rather than optimizing only for one.

---

## 5. Key Takeaway

Transformer chips exist because attention's data-dependent, growing-KV-cache access pattern breaks the assumptions that make dense-matmul tensor cores efficient. The winning architectures fuse the score–softmax–weighted-sum pipeline to avoid materializing the quadratic attention matrix, quantize the KV cache aggressively to cut the dominant bandwidth cost of decoding, and size memory and batching separately for the compute-bound prefill phase and the bandwidth-bound decode phase. Throughput on real transformer serving is set by how well these three choices fit the hardware's actual memory bandwidth — not by peak FLOPs.

---

## References

  • Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," NeurIPS 2022.
  • NVIDIA, "FP8 Formats for Deep Learning" and Transformer Engine documentation.
  • Pope et al., "Efficiently Scaling Transformer Inference," MLSys 2023 — KV cache and batching trade-offs.
  • Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers."

Take Transformer chips further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app