Parallel File Io

Parallel File I/O — reading and writing data across multiple storage devices and processes simultaneously, essential for HPC and large-scale data processing where sequential I/O is a bottleneck.

Why Parallel I/O?

• Single disk: ~200 MB/s sequential read
• 100 disks in parallel: ~20 GB/s → 100x faster
• Large-scale simulations and AI training generate/consume TB–PB of data

Parallel Filesystems

• Lustre: Most common HPC filesystem. Separates metadata (MDS) from data (OSS). Scales to 1000s of clients, PB+ storage, 1+ TB/s aggregate bandwidth
• GPFS/Spectrum Scale (IBM): Enterprise parallel filesystem. Strong metadata performance
• BeeGFS: Open-source, easy to deploy. Popular for AI clusters
• WekaIO: Flash-native parallel filesystem. Ultra-low latency

Striping

• Files split into chunks distributed across storage servers
• Client reads/writes to multiple servers in parallel
• Stripe size: 1-4 MB typical. Tunable for workload

Parallel I/O Libraries

• MPI-IO: Part of MPI standard. Collective I/O for coordinated access
• HDF5: Self-describing scientific data format. Parallel HDF5 for multi-process access
• NetCDF: Climate/weather data. Parallel variant available
• POSIX I/O: Not parallel-aware → contention at filesystem level

Best Practices

• Large sequential writes >> many small random writes
• Use collective I/O (aggregate small requests into large ones)
• Match stripe count to number of writing processes

Parallel I/O is often the overlooked bottleneck — a perfectly parallelized computation means nothing if data loading/saving can't keep up.