HBM Parallel
High-bandwidth memory (HBM) is DRAM stacked vertically and connected to its host through an interface thousands of bits wide, running beside the processor on an interposer rather than on a motherboard. It exists because conventional DRAM interfaces cannot feed a modern accelerator: bandwidth is the product of interface width and per-pin rate, and only one of those can be pushed indefinitely.
A DDR5 channel is 64 bits wide and must drive centimetres of motherboard trace, so the per-pin rate is pushed hard and the width stays small. HBM inverts the trade: by placing the stack microns away across an interposer, each wire is short, low-capacitance and low-energy, so the bus can be made enormously wide at a modest clock.
The generational progression
| Generation | Bus width | Per-pin rate | Stack bandwidth | Stack height |
|---|---|---|---|---|
| HBM2E | 1024-bit | 3.2 Gb/s | ~410 GB/s | 8-Hi |
| HBM3 | 1024-bit | 6.4 Gb/s | ~819 GB/s | 8/12-Hi |
| HBM3E | 1024-bit | 9.2–9.6 Gb/s | ~1.2 TB/s | 8/12-Hi |
| HBM4 | 2048-bit | ~6.4–8 Gb/s | ~1.6–2.0 TB/s | 12/16-Hi |
HBM4's defining change is the doubling of the interface to 2048 bits per stack. Note what it does to the per-pin rate: it comes *down* relative to HBM3E. That is deliberate. Signal integrity and energy per bit both degrade superlinearly with pin rate, so widening the bus and relaxing the clock delivers more bandwidth at lower energy — provided the physical interface can carry twice the wires.
Why interconnect pitch is the real constraint
Doubling the bus means doubling the connections under the stack, and that is a packaging problem, not a DRAM problem. Micro-bumps, the incumbent, are solder joints on a pitch that has stalled near 40 to 45 μm: solder volume cannot shrink indefinitely without bridging, and the intermetallic that forms on reflow becomes a growing fraction of a shrinking joint.
Direct bond interconnect replaces the solder entirely. Copper pads on the two surfaces are planarised and bonded metal-to-metal, with the surrounding dielectric bonding at the same time, giving pitches below 10 μm and on a path below 1 μm. The interconnect density available scales as the inverse square of pitch:
so moving from a 45 μm micro-bump to a 9 μm hybrid bond is not a five-fold improvement but a twenty-five-fold one. That is the headroom HBM4 spends on its 2048-bit bus.
The base logic die
Every stack sits on a base die that aggregates the TSV columns, runs the PHY, and handles test and repair. Through HBM3E this die was built on a mature DRAM-like process. HBM4 moves it to an advanced logic node — FinFET or nanosheet, fabricated by a logic foundry — because a 2048-bit PHY running at multi-gigabit rates is a logic problem that a DRAM process cannot serve efficiently.
This is the structural change with the longest consequences. Once the base die is a logic die, it can absorb function that used to live on the host: scheduling, ECC, compression, and eventually arithmetic close to the data. It also means an HBM stack is now a heterogeneous assembly spanning two foundries and two process families, with the commercial and yield coupling that implies.
Where it binds in LLM inference
Transformer decoding is memory-bound, not compute-bound. Generating one token requires streaming the entire weight set and the accumulated KV cache through the arithmetic units, and each weight is used for a single multiply-accumulate before being discarded. The achievable token rate is therefore set by bandwidth, not FLOPs:
with the KV cache growing linearly in context length and batch size:
A model whose weights occupy 140 GB, served from 3.2 TB/s of aggregate HBM, cannot exceed roughly 23 ms per token per replica however much compute sits beside it. Long contexts make it worse, because the KV term grows without bound while the weight term is fixed. This is why accelerator roadmaps are quoted in TB/s rather than TFLOPs, why capacity per stack matters as much as bandwidth, and why HBM supply — not lithography — has been the binding constraint on AI deployment.