Interconnect Topology
Interconnect Topology is the physical and logical arrangement of network links connecting compute nodes in parallel systems — determining the bandwidth, latency, scalability, and cost characteristics of the communication fabric that enables thousands to millions of processors to work together, with topology choice directly impacting application performance by 2-5x for communication-heavy workloads.
Common Topologies
| Topology | Bisection BW | Diameter | Cost (links) | Used By |
|---|---|---|---|---|
| Fat Tree | Full bisection | 2 log N | O(N log N) | Ethernet clusters, InfiniBand |
| 3D Torus | O(N^(2/3)) | O(N^(1/3)) | O(N) | IBM Blue Gene, Fugaku |
| Dragonfly | ~Full bisection | 3-5 hops | O(N^(4/3)) | Cray XC/Slingshot |
| Hypercube | O(N) | log N | O(N log N) | Historical (CM-2) |
| Mesh (2D/3D) | O(√N) | O(√N) | O(N) | GPU NVSwitch mesh |
Fat Tree (Clos Network)
- Structure: Multi-level tree with increasing bandwidth at each level → "fat" at top.
- Full bisection bandwidth: Any half of nodes can communicate with other half at full speed.
- Implementation: Standard Ethernet/InfiniBand switches in leaf-spine-core layers.
- Pros: Non-blocking, any-to-any communication at full bandwidth.
- Cons: Expensive — top-level switches carry all traffic. Cable count: O(N log N).
3D Torus
- Structure: Each node connected to 6 neighbors (±x, ±y, ±z). Wrap-around links at edges.
- IBM Blue Gene/Q: 5D torus with 10 links per node.
- Fujitsu Fugaku (#1 in 2020): 6D mesh/torus (Tofu-D interconnect).
- Pros: Simple, low cost (O(N) links), good for nearest-neighbor communication (stencil patterns).
- Cons: Low bisection bandwidth — all-to-all communication suffers.
Dragonfly
- Structure: Three levels — intra-group (local), inter-group (global), inter-cabinet.
- Groups: Fully connected internally. Groups connected by global links.
- Adaptive routing: Traffic dynamically routed to avoid congestion.
- Cray Slingshot / HPE Cray EX: Modern HPC systems use Dragonfly variants.
- Pros: Good balance of cost, bandwidth, and latency for diverse traffic patterns.
NVLink/NVSwitch Topology (GPU clusters)
- DGX A100 (8 GPUs): Full NVSwitch mesh — any GPU to any GPU at 600 GB/s.
- DGX H100 (8 GPUs): NVSwitch 4th gen — 900 GB/s per GPU.
- NVLink Network (multi-node): NVLink extended across nodes → GPU-to-GPU without CPU.
Routing Algorithms
- Deterministic: Same path for same source-destination → simple, may cause congestion.
- Adaptive: Route based on current congestion → better utilization, harder to implement.
- Minimal: Shortest path only. Non-minimal: May take longer paths to avoid congestion.
Interconnect topology is a defining architectural choice for any large-scale parallel system — the topology determines the communication performance envelope within which all parallel algorithms must operate, making it one of the first and most consequential decisions in supercomputer and data center design.