TPU
A Tensor Processing Unit (TPU) is Google's custom machine-learning accelerator, designed around high-throughput matrix multiplication for training and inference in the Google software and cloud ecosystem.
The core is a systolic matrix unit. Activations and weights stream through a two-dimensional multiply-accumulate array, maximizing data reuse and reducing energy spent moving operands. Vector units and on-chip memory handle the operations surrounding matrix multiplication.
Why the systolic array is so efficient. In a conventional processor every multiply re-fetches its operands from registers or memory. In a systolic array a weight is loaded once into a cell and stays put, reused across a whole stream of activations, while partial sums accumulate as they march down the array. One memory read therefore feeds many multiply-accumulates, and that operand reuse is where the energy-per-operation advantage comes from.
TPUs are systems, not isolated chips. High-bandwidth memory, fast chip-to-chip interconnects, and pod-scale topology allow large models to span many devices. XLA compiles TensorFlow, JAX, and supported PyTorch workloads into programs matched to that hardware.
The generations trace a clear arc. TPU v1 in 2016 was inference-only, with an 8-bit matrix unit on a PCIe card. v2 and v3 added training, HBM, and the bfloat16 numeric format, and organized chips into pods. v4 introduced optically reconfigurable interconnects that rewire the pod topology to the workload, and the later v5e and v5p chase performance per dollar and per watt at very large scale. Throughout, the real differentiator is the pod: thousands of chips wired into a single machine.
The trade is specialization for efficiency. TPUs can be highly efficient on well-shaped tensor workloads, while GPUs generally offer a broader programming ecosystem and more flexibility for irregular kernels.
| Generation | Focus | Notable |
|---|---|---|
| TPU v1 (2016) | inference only | 8-bit MXU on a PCIe card |
| TPU v2 / v3 | training and inference | HBM, bfloat16, pod scale |
| TPU v4 | large-scale training | optical circuit switch interconnect |
| TPU v5e / v5p | scale and cost | performance per dollar, huge pods |
Read a TPU through an *operand-reuse* lens rather than a *peak-FLOPS* lens: the systolic array wins by getting many multiply-accumulates out of each value it reads from memory, so its efficiency comes from dataflow — how long a weight stays resident and how far a partial sum travels before it is written back — not from clock speed. Every design choice around it, from bfloat16 to pod-scale optical interconnects, is about keeping that array fed without paying to move data.