Artificial Intelligence
Artificial intelligence, in the sense that matters to a chip company, is a family of statistical methods that learn a function from data instead of having it specified by a programmer — together with the accelerator hardware built because those methods turn out to be dominated by dense matrix arithmetic. The relationship runs both ways: the methods set the economics of the hardware, and the hardware decides which methods anyone can afford to run.
The two halves of that diagram are the key to the whole field. Training is one enormous batch job measured in GPU-months; inference is a latency-sensitive service measured in cost per million tokens. They stress completely different parts of the hardware, and almost every design decision in an AI accelerator is a compromise between them.
Learning is curve-fitting with an extraordinary number of knobs. A model is a parameterised function; training is the search for parameter values that minimise error on data. Deep learning does this with layered networks and backpropagation, which is nothing more exotic than the chain rule applied efficiently to compute how every one of billions of parameters should move. What makes it powerful is not the optimiser but the fact that the useful features are learned rather than hand-engineered: no one wrote down what an edge detector or a syntactic dependency should look like, and that is precisely why the approach generalised across vision, speech and language.
Training cost is arithmetic you can estimate on a napkin. For a dense transformer, the compute required is roughly six floating-point operations per parameter per training token — two for the forward pass and about four for the backward pass:
A 70-billion-parameter model trained on 2 trillion tokens therefore needs about $8.4 \times 10^{23}$ FLOPs. At a realistic sustained 400 TFLOP/s per accelerator — roughly 40 percent utilisation of a peak BF16 rate — that is about 580,000 accelerator-hours, or a cluster of 2,400 devices running for ten days. This equation is why frontier model development is a capital-expenditure business and why accelerator supply, not algorithmic insight, has been the practical constraint on who can train at the frontier.
Inference is a bandwidth problem, not a FLOPs problem. Generating text autoregressively means reading every weight in the model from memory to produce each token. A 70-billion-parameter model in BF16 is 140 GB of weights; at 3 TB/s of HBM bandwidth that is about 47 ms per token, a ceiling of roughly 21 tokens per second — and the arithmetic units sit nearly idle while that traffic completes. Nothing about adding FLOPs helps. What helps is moving fewer bytes or reusing each byte more: batching many requests so one weight read serves many tokens, caching keys and values so earlier context is not recomputed, and reducing the precision of the weights themselves.
Which is why numeric precision became an architectural decision. The progression from FP32 through TF32, BF16, FP8 and now FP4 is not a numerical-analysis footnote; each halving of the format halves the bytes moved and roughly doubles achievable dense throughput. Quantisation to FP8 or INT4, structured pruning, and distillation into a smaller student model are all the same move in different clothing — trading a small, measurable amount of output quality for a large, immediate reduction in memory traffic. On the serving side of the business, that trade is usually overwhelmingly worth making.
| Workload | Dominant resource | Usual bottleneck | Main lever |
|---|---|---|---|
| Pre-training | Dense FLOPs | Interconnect and cluster scale | More accelerators, better parallelism |
| Fine-tuning | FLOPs and memory capacity | Optimiser state footprint | LoRA and adapter methods |
| Batch inference | Memory bandwidth | Weight reads per token | Batching, quantisation |
| Interactive inference | Latency | KV-cache size and growth | Caching, speculative decoding |
The accelerator is itself a semiconductor product, which closes the loop. A modern training GPU is a reticle-limit-sized die — often several dies — on a CoWoS silicon interposer surrounded by HBM stacks, drawing 700 to 1400 W, linked to its neighbours by NVLink or InfiniBand, and cooled by liquid because air cannot remove that much heat from a 130 kW rack. Everything the infrastructure layer below this one is concerned with — advanced nodes, packaging capacity, HBM supply, power delivery — exists in its current form largely because of AI demand. And the loop runs the other way too: AI methods are now applied back to chip design and manufacturing, in GPU-accelerated inverse lithography, ML defect classification, and automated place-and-route optimisation.
That cycle is the reason an AI article belongs in a semiconductor hierarchy at all. AI is not merely a customer of chips; it is currently the single largest force shaping which transistors, which memory, and which packaging get built.
Next in the stack are large language models. They are one branch of the deep learning methods described here — distinctive enough in architecture, scale and deployment economics to be worth treating separately, and currently the branch driving most of the accelerator demand described above.
Read artificial intelligence through a *bytes-moved* lens rather than a *FLOPs* lens. Training is bounded by total arithmetic and is therefore a capital problem, estimable in advance from $C \approx 6ND$. Everything after training is bounded by how much data has to cross the memory interface per useful token, which is why HBM bandwidth, quantisation, batching and KV-cache management dominate the economics of deployed AI far more than peak TFLOP/s on a datasheet. The headline FLOPs number tells you what a chip could do; the bandwidth and the precision tell you what it will actually deliver, and what each answer costs to produce.