Parallel Io File System

Parallel I/O and File Systems are the storage infrastructure that enables thousands of compute nodes to simultaneously read and write data at aggregate bandwidths of hundreds of GB/s to TB/s — using parallel file systems (Lustre, GPFS/Spectrum Scale, BeeGFS) that stripe data across hundreds of storage servers and expose POSIX or MPI-IO interfaces, because HPC and AI workloads that generate petabytes of data per simulation run would bottleneck on serial I/O by orders of magnitude.

Why Parallel I/O

A single storage server provides 1-5 GB/s sequential bandwidth. A supercomputer with 10,000 nodes running a climate simulation writes a 100 TB checkpoint every hour — requiring 28+ GB/s sustained. Only parallel I/O across many storage servers can achieve this.

Parallel File System Architecture

• Lustre (Open-Source, most widely deployed in HPC):
• MDS (Metadata Server): Handles file creation, directory operations, permissions. One active MDS per filesystem (HA pair). Bottleneck for metadata-heavy workloads (many small files).
• OSS (Object Storage Server): Manages one or more OSTs (Object Storage Targets — typically RAID groups of disks or SSDs). Data is striped across OSTs for bandwidth.
• Client: Mounts the filesystem on compute nodes. Reads/writes go directly to OSTs (no MDS in the data path). Striping: a single file is divided into stripes distributed round-robin across OSTs.
• Performance: Top installations achieve 1+ TB/s aggregate bandwidth with 100+ OSTs.

• GPFS/Spectrum Scale (IBM):
• Shared-disk model — all nodes access all disks through a SAN.
• Distributed locking (token-based) for concurrent access.
• Integrated tiering: hot data on SSDs, warm on HDDs, cold archived to tape.
• Used in: enterprise HPC, AI training clusters, financial services.

MPI-IO

The parallel I/O interface for MPI programs:
- Collective I/O: All processes in a communicator participate in a single I/O operation. The MPI library aggregates small, scattered requests into large, sequential I/O operations — dramatically improving effective bandwidth.
- Two-Phase I/O: Phase 1: shuffle data among processes so that each process handles a contiguous range of the file. Phase 2: each process reads/writes its contiguous range. Converts random I/O to sequential.
- File Views: Each process defines its view (subarray, distribution) of a shared file using MPI datatypes. MPI-IO uses views to compute optimal aggregation.

I/O Optimization Techniques

• Striping Configuration: Match stripe count and stripe size to workload. Large sequential writes benefit from high stripe count (all OSTs active). Small random I/O benefits from lower stripe count (reduce metadata overhead).
• Burst Buffer: NVMe tier between compute nodes and parallel file system. Absorbs checkpoint writes at 100+ GB/s, drains to disk in the background. NERSC Perlmutter uses 30 PB Lustre + 1.5 PB NVMe burst buffer.
• I/O Forwarding: Reduce the number of nodes accessing the file system simultaneously. I/O forwarding nodes aggregate requests from compute nodes, presenting fewer concurrent clients to the file system.

Parallel I/O and File Systems are the data infrastructure backbone of HPC and AI — providing the bandwidth to feed data-hungry simulations and training runs, and the capacity to store the petabytes of results that drive scientific discovery and model development.