Parallel Io Storage
Parallel I/O and Storage Systems are the hardware and software architectures that enable multiple processes to simultaneously read and write data to storage — essential for HPC and data-intensive computing where sequential I/O becomes the bottleneck, with parallel file systems like Lustre and GPFS distributing data across hundreds of storage servers to deliver aggregate bandwidths of terabytes per second.
The I/O Bottleneck
- Modern HPC: Thousands of cores computing at PFLOPS → need TB/s of data I/O.
- Single SSD: ~7 GB/s (NVMe Gen5). Single HDD: ~200 MB/s.
- 10,000 processes checkpoint at once → need 100+ GB/s aggregate.
- Without parallel I/O: Processes serialize at storage → compute idle → massive waste.
Parallel File Systems
| System | Developer | Stripe Unit | Max Bandwidth | Typical Use |
|---|---|---|---|---|
| Lustre | OpenSFS | Object Storage Targets (OSTs) | 1+ TB/s | HPC, national labs |
| GPFS/Spectrum Scale | IBM | Network Shared Disk | 1+ TB/s | Enterprise HPC |
| BeeGFS | ThinkParQ | Storage targets | 100+ GB/s | Research clusters |
| CephFS | Red Hat | RADOS objects | 100+ GB/s | Cloud, HPC |
| DAOS | Intel | SCM + NVMe | High IOPS + bandwidth | Exascale HPC |
Lustre Architecture
- MDS (Metadata Server): Handles namespace operations (open, stat, mkdir).
- OSS (Object Storage Server): Serves data to/from OSTs.
- OST (Object Storage Target): Physical storage (disk arrays) — typically dozens to hundreds.
- Striping: File split across multiple OSTs — each process reads from different OST simultaneously.
- Stripe size: Configurable (typically 1-4 MB).
- Stripe count: Number of OSTs per file (8-64 typical for large files).
MPI-IO (Standard Parallel I/O API)
- Extends MPI with file I/O operations: `MPI_File_open`, `MPI_File_read_all`, `MPI_File_write_at`.
- Collective I/O: All processes coordinate I/O → library optimizes access pattern.
- Two-phase I/O: Aggregate small scattered accesses into large sequential accesses → 10-100x speedup.
- Data sieving: Read a contiguous chunk, extract needed data → reduces number of I/O operations.
HDF5 Parallel
- High-level data format for scientific data (hierarchical, self-describing).
- Parallel HDF5: Built on MPI-IO — multiple processes write to same HDF5 file simultaneously.
- Chunking: Dataset divided into chunks — different processes write different chunks.
- Compression: Per-chunk compression — parallel decompression possible.
I/O Optimization Strategies
| Strategy | How | Benefit |
|---|---|---|
| Increase stripe count | File spread across more OSTs | Higher aggregate bandwidth |
| Align to stripe boundary | Process I/O aligned to stripe units | Fewer cross-OST operations |
| Collective I/O | Coordinate processes → merge small I/Os | Reduce metadata + seek overhead |
| Burst buffer | Fast tier (NVMe) absorbs bursts | Smooth out I/O peaks |
| Asynchronous I/O | Overlap I/O with computation | Hide I/O latency |
Parallel I/O and storage systems are the critical data backbone for computational science — without them, the raw computing power of modern supercomputers would be stranded waiting for data, making parallel storage architecture as important as the compute architecture for overall system performance.