AI

Artificial intelligence, in the sense that matters to anyone building silicon, is a surprisingly narrow computational shape: dense linear algebra, repeated at enormous scale, over numbers that turn out not to need much precision. Nearly everything distinctive about modern accelerators — systolic arrays, tensor cores, high-bandwidth memory, eight-bit floating point — follows from that one observation and from the fact that memory cannot keep up with arithmetic.

Almost all of the arithmetic is one operation. In a transformer, well over ninety percent of the floating-point work is matrix multiplication — and convolutions in vision models are lowered to the same primitive. That concentration is unusual and enormously convenient: a processor does not need to be good at general computation to be good at this, it needs to be good at multiplying and accumulating in bulk, with operands reused enough to avoid fetching them repeatedly.

The shape of the workload is why the hardware looks the way it does. A general-purpose core spends most of its transistors on control — branch prediction, out-of-order scheduling, cache coherence — because it cannot know what comes next. A matrix multiply has no such uncertainty, so an accelerator spends its area on multiply-accumulate units and on getting operands to them. A systolic array takes this to its conclusion: a grid of MACs through which operands flow in lockstep, each one reused across an entire row and column before it leaves. Tensor cores are the same bargain in a different package — an instruction that performs a small matrix multiply as a single operation rather than a sequence of scalar ones.

Whether you are compute-bound or memory-bound is the first question, and usually the last. Arithmetic intensity is FLOPs performed per byte moved. Every machine has a balance point where its peak compute and its peak bandwidth meet — on a current accelerator, roughly three hundred FLOPs per byte. Above that, a kernel can in principle reach peak FLOPS. Below it, the arithmetic units idle while memory catches up, and adding compute is money burned. Large-batch GEMM sits comfortably above the balance point. Autoregressive decoding sits near two FLOPs per byte, because generating a single token requires reading every weight in the model, and reuses each one exactly once. That is why inference serving is a memory-bandwidth business dressed up as a compute business, and why batching matters so much: it is the only way to reuse a weight once it has been fetched.

Precision is the cheapest lever anyone has. Halving the width of a number halves the bytes moved, halves the memory it occupies, and roughly doubles how many multiply-accumulate units fit in the same silicon area. The industry has walked steadily down that ladder — FP32 to TF32 to BF16 for training, then FP8, and now four-bit block-scaled formats — and each step has held up better than expected. The reason BF16 displaced FP16 is instructive: it has fewer mantissa bits, not more, but it retains FP32's exponent range, and neural network training turns out to be far more sensitive to dynamic range than to precision. Inference goes further still, because a trained network can be quantised to eight or four bits with careful scaling and lose very little accuracy.

Sparsity has promised more than it has delivered. Most weights in a trained network contribute little, so zeroing them ought to be free performance. In practice, unstructured sparsity produces irregular memory access that defeats the very regularity accelerators exploit, so the gains evaporate. Structured schemes such as two-non-zeros-in-every-four are hardware-friendly and deliver a real but bounded speedup. Mixture-of-experts is the form of sparsity that has genuinely worked at scale, because it skips whole blocks of computation rather than scattered individual weights.

The memory wall is the constraint everything else is arranged around. Across successive generations, peak arithmetic throughput has grown far faster than the bandwidth available to feed it. Both improved; they did not improve together. High-bandwidth memory is the direct response — DRAM stacked vertically and placed beside the die on an interposer, bought at real cost in packaging complexity because no conventional memory interface can supply what the arithmetic can consume. It narrows the gap rather than closing it, which is why on-chip SRAM capacity, operator fusion and cache-aware scheduling all remain worth large engineering investments.

Co-design is now the normal way progress happens. FlashAttention is the canonical example: it computes exactly the same attention as the naive implementation, but tiles the work so intermediate results stay in on-chip SRAM instead of being written out to HBM and read back. No new mathematics, no accuracy change — purely a rearrangement to respect the memory hierarchy, and a large speedup. The same pattern recurs throughout: quantisation-aware training exists because hardware wants eight-bit operands, mixture-of-experts exists partly because dense scaling outruns memory, and grouped-query attention exists because the KV cache became the bottleneck in serving.

At scale, an AI system stops being a chip problem. A model too large for one device is split three ways at once: data parallelism replicates it across devices and averages gradients, tensor parallelism splits individual matrices across devices within a node, and pipeline parallelism assigns different layers to different nodes. Each split introduces communication — all-reduce, all-gather, point-to-point activations — and the interconnect becomes as decisive as the arithmetic. This is where accelerator design meets the datacenter, and why a cluster's topology is part of its performance specification rather than a facilities detail.

Read AI through an *intensity* lens rather than a *FLOPS* lens: the headline number on an accelerator's datasheet is reachable only by kernels that reuse their operands heavily, and a great deal of real work does not. Ask how many arithmetic operations a kernel performs per byte it must move, compare that to the machine's balance point, and the entire design conversation — precision, batching, fusion, memory technology, interconnect — follows from which side of the line you land on.