PHY Controller HBM

High-bandwidth memory (HBM) is DRAM stacked vertically and connected to its host through an interface thousands of bits wide, running beside the processor on an interposer rather than on a motherboard. It exists because conventional DRAM interfaces cannot feed a modern accelerator: bandwidth is the product of interface width and per-pin rate, and only one of those can be pushed indefinitely.

A DDR5 channel is 64 bits wide and must drive centimetres of motherboard trace, so the per-pin rate is pushed hard and the width stays small. HBM inverts the trade: by placing the stack microns away across an interposer, each wire is short, low-capacitance and low-energy, so the bus can be made enormously wide at a modest clock.

The generational progression

HBM4's defining architectural change is the doubling of the interface to 2048 bits per stack. The original 8 Gb/s baseline could deliver about 2 TB/s without demanding a faster pin than HBM3E, but commercial implementations have since pushed beyond 10 Gb/s: current supplier disclosures span more than 2.8 TB/s to 3.3 TB/s per stack. Width created the first gain; improved PHY design is now compounding it. The controller must sustain enough bank-level parallelism to feed all of those pins rather than merely advertise their peak rate.

Why interconnect pitch is the real constraint

Doubling the bus means doubling the connections under the stack, and that is a packaging problem, not a DRAM problem. Micro-bumps, the incumbent, are solder joints on a pitch that has stalled near 40 to 45 μm: solder volume cannot shrink indefinitely without bridging, and the intermetallic that forms on reflow becomes a growing fraction of a shrinking joint.

Direct bond interconnect replaces the solder entirely. Copper pads on the two surfaces are planarised and bonded metal-to-metal, with the surrounding dielectric bonding at the same time, giving pitches below 10 μm and on a path below 1 μm. It is a scaling path for later, taller HBM generations rather than a requirement for every HBM4 product: current HBM4 stacks can still use refined thermocompression or molded-underfill assembly. The interconnect density available scales as the inverse square of pitch:

so moving from a 45 μm micro-bump to a 9 μm hybrid bond is not a five-fold improvement but a twenty-five-fold one. That density is roadmap headroom for taller stacks and still-wider interfaces; the 2048-bit HBM4 interface does not, by itself, prove that hybrid bonding is present.

The base logic die

Every stack sits on a base die that aggregates the TSV columns, runs the PHY, and handles test and repair. Through HBM3E this die was built on a mature DRAM-like process. HBM4 moves it to an advanced logic node — FinFET or nanosheet, fabricated by a logic foundry — because a 2048-bit PHY running at multi-gigabit rates is a logic problem that a DRAM process cannot serve efficiently.

This is the structural change with the longest consequences. Once the base die is a logic die, it can absorb function that used to live on the host: scheduling, ECC, compression, and eventually arithmetic close to the data. It also means an HBM stack is now a heterogeneous assembly spanning two foundries and two process families, with the commercial and yield coupling that implies.

Where it binds in LLM inference

Transformer decoding is memory-bound, not compute-bound. Generating one token requires streaming the entire weight set and the accumulated KV cache through the arithmetic units, and each weight is used for a single multiply-accumulate before being discarded. The achievable token rate is therefore set by bandwidth, not FLOPs:

with the KV cache growing linearly in context length and batch size:

A model whose weights occupy 140 GB, served from 3.2 TB/s of aggregate HBM, cannot exceed roughly 23 ms per token per replica however much compute sits beside it. Long contexts make it worse, because the KV term grows without bound while the weight term is fixed. This is why accelerator roadmaps are quoted in TB/s rather than TFLOPs, why capacity per stack matters as much as bandwidth, and why HBM supply — not lithography — has been the binding constraint on AI deployment.

That $BW_{\text{mem}}$ in the token-time equation is a ceiling, not a guarantee — closing the gap to it is the memory controller's entire job. The controller sitting between the accelerator's compute units and the HBM4 PHY has to turn a stream of independent read and write requests into a command sequence that actually keeps every channel busy, and a naive scheduler leaves most of the physical bandwidth on the table.

How well that works comes down almost entirely to whether the address mapping spreads traffic across banks and channels or concentrates it. Define bandwidth efficiency as the fraction of theoretical peak actually sustained:

A naive mapping that assigns consecutive addresses to the same bank turns any sequential-ish access pattern into a chain of bank conflicts — every request has to wait for the previous row to close before the next one can open — and $\eta$ collapses toward the single-bank case no matter how many channels HBM4 provides. Interleaving the channel, bank, and pseudo-channel bits *into* the low-order address bits, often with an XOR hash rather than a plain bit slice, spreads consecutive and strided accesses across independent banks instead, so most requests find their target bank already open or at least idle.

That is the real reason the token-time equation above treats $BW_{\text{mem}}$ as a single number rather than something the workload negotiates. HBM4's 2048-bit interface and taller stacks only raise $BW_{\text{peak}}$; whether an actual inference workload gets anywhere close to it depends on how well its access pattern is distributed across the channels and banks the controller can keep active. Mapping, scheduling, queue depth and workload locality jointly set the achieved result—no mapping can guarantee ideal efficiency for every access stream.

This controller is also the commercial toll bridge between expensive memory and useful accelerator throughput. The defensible asset is not one scheduler rule; it is the verified combination of address hashing, timing closure, PHY training, repair, ECC, thermal throttling and workload-aware quality of service. A weak controller can strand bandwidth already paid for in silicon, packaging and HBM supply, while a strong one converts the same physical stack into more tokens per second.