Io Parallelism
Parallel I/O and File Systems encompasses the techniques and systems for achieving high-bandwidth storage access by distributing file data (striping) across multiple storage servers and disks, enabling concurrent I/O from hundreds to thousands of compute nodes — essential for scientific computing, AI training, and large-scale data analytics where I/O bottlenecks can dominate total application time.
The I/O Bottleneck: A single disk provides ~200 MB/s sequential bandwidth. A single NVMe SSD provides ~7 GB/s. But a large HPC application running on 1000 nodes may need 1+ TB/s aggregate bandwidth. Parallel file systems achieve this by striping data across thousands of storage targets.
Parallel File System Architecture:
| Component | Function | Example |
|---|---|---|
| Metadata servers (MDS) | Directory ops, file attributes | Lustre MDT, GPFS mmfsd |
| Object storage servers (OSS) | Store file data stripes | Lustre OST, BeeGFS storage |
| Clients | Parallel access from compute nodes | POSIX client, FUSE mount |
| Network | High-bandwidth interconnect | InfiniBand, 100GbE |
File Striping: A file is divided into fixed-size chunks (stripe units, typically 1-4 MB) distributed round-robin across multiple storage targets. A file with 1 MB stripe across 100 OSTs gets 100x the bandwidth of a single OST. Stripe count and size are tunable per file or directory — small files benefit from low stripe count (avoids metadata overhead), large files from high stripe count.
Key Systems:
- Lustre: Dominant HPC parallel file system. Separates metadata (MDS) from data (OSS). Supports file-level striping with progressive file layouts. Scalable to exabyte capacity with thousands of OSTs. POSIX-compliant.
- GPFS/Spectrum Scale (IBM): Block-level distributed file system with shared-disk architecture. Every node can be both client and server. Strong consistency, high metadata performance. Common in enterprise HPC and AI.
- BeeGFS: Lightweight parallel file system with separate metadata and storage servers. Easy deployment. Gaining adoption in AI/ML clusters.
- DAOS: Intel's next-generation storage system targeting NVMe and persistent memory. Bypasses kernel and POSIX for lowest latency. Designed for exascale.
MPI-IO: The standard parallel I/O interface for HPC applications. Key concepts: collective I/O — processes coordinate I/O operations to merge many small requests into fewer large ones (two-phase I/O); file views — each process describes its access pattern using MPI datatypes, enabling non-contiguous I/O without performance loss; hints — tuning parameters (striping, buffering, aggregator count) communicated through MPI_Info objects.
I/O Optimization Strategies: Aggregate small I/O into large writes (collective I/O, buffered I/O); align access to stripe boundaries to avoid lock contention; use dedicated I/O nodes (burst buffers, I/O forwarding) to absorb bursty write patterns; data staging — stage checkpoint data to fast local NVMe before flushing to parallel FS; and avoid metadata storms — directory listing or stat() from thousands of nodes simultaneously overwhelms MDS.
Parallel I/O is the often-overlooked third leg of HPC performance (alongside compute and communication) — a perfectly optimized computation that writes results through a serial I/O bottleneck wastes all the parallelism it worked so hard to achieve.