Tensor Parallelism
Tensor parallelism is a model-parallel technique that splits the individual weight matrices inside a layer across multiple GPUs, so each device stores and multiplies only a slice of the layer rather than the whole thing. Where the model is too large for one accelerator's memory or too slow on one device, tensor parallelism divides each matmul — the dominant operation in a transformer — into shards computed in parallel, then recombines the partial results with a collective communication step before the next layer runs.
It shards the matmul, not the model's layers. A layer computes Y = X·W. Tensor parallelism cuts the weight matrix W into pieces — by columns or by rows — and hands each piece to a different GPU. Every GPU multiplies the same input by its own shard, producing a partial slice of the output. Because the work is divided within a single operation, all participating GPUs run the same layer at the same time on different data columns, which is why it is called intra-layer parallelism, in contrast to pipeline parallelism that assigns whole layers to different devices.
The unavoidable cost is a collective every layer. Splitting W means no single GPU holds the full output, so the shards must be stitched back together — an all-reduce (for row splits) or all-gather (for column splits) — on every forward pass and again on every backward pass. That collective moves activation-sized data between all GPUs in the group, and it sits on the critical path: the next layer cannot start until the recombination finishes. As the group grows, per-GPU compute shrinks toward 1/N but the communication share climbs, so the interconnect, not the math, sets the ceiling.
| Tensor parallelism | Pipeline parallelism | |
|---|---|---|
| Splits | weights within a layer | whole layers into stages |
| Granularity | intra-layer | inter-layer |
| Communication | all-reduce/all-gather per layer | activations at stage boundaries |
| Frequency | every layer, fwd + bwd | between stages |
| Best domain | fast NVLink island | across nodes tolerable |
It only pays off inside a fast interconnect domain. Because a collective fires on every sharded layer, tensor parallelism is bandwidth- and latency-bound and is normally confined to the GPUs wired together by a high-speed scale-up fabric such as NVLink within a single server. Push it across slower links between nodes and the all-reduce dominates, erasing the compute speedup. In practice large models combine it with the others: tensor parallelism inside a node, pipeline parallelism across nodes, and data parallelism across replicas — each chosen to match the bandwidth available at that level.
Read tensor parallelism through a quant lens rather than a 'just add GPUs' lens: the useful compute per GPU falls as 1/N, but the collective moves activation-sized bytes every layer, so the speedup holds only while interconnect bandwidth keeps the all-reduce shorter than the compute it overlaps. The design question is the ratio of matmul FLOPs to bytes-per-collective at a given N — which is exactly why tensor parallelism lives inside an NVLink island and stops at its edge, where the communication term overtakes the compute term.