Distributed File System
Distributed and Parallel File Systems (Lustre, HDFS, GPFS) are the storage systems that stripe files across many servers to provide aggregate I/O bandwidth and capacity far beyond any single storage node — essential for HPC simulations that read/write terabytes of checkpoint data, ML training pipelines that stream petabytes of training data, and analytics workloads that process massive datasets, where parallel I/O at 100+ GB/s is a fundamental requirement.
Why Distributed File Systems
- Single NFS server: ~1-5 GB/s, ~100 TB capacity. Insufficient for HPC/ML workloads.
- Distributed FS: Aggregate bandwidth scales with number of servers.
- 100 storage servers × 5 GB/s each = 500 GB/s aggregate → reads entire dataset in seconds.
Major Systems Comparison
| System | Developer | Primary Use | Max Performance |
|---|---|---|---|
| Lustre | OpenSFS | HPC, supercomputing | 1+ TB/s |
| GPFS/Spectrum Scale | IBM | Enterprise HPC | 500+ GB/s |
| HDFS | Apache | Big data (Hadoop/Spark) | 100+ GB/s |
| BeeGFS | ThinkParQ | ML training, HPC | 200+ GB/s |
| CephFS | Red Hat | Cloud, general purpose | 100+ GB/s |
| WekaFS | WEKA | ML training (NVMe-native) | 300+ GB/s |
Lustre Architecture
- MDS (Metadata Server): Handles file names, directories, permissions.
- OSS (Object Storage Server): Serves file data chunks.
- OST (Object Storage Target): Physical storage volumes.
- Striping: Large files split across multiple OSTs → parallel I/O.
HDFS Architecture
- NameNode: Single metadata server (directories, block locations).
- DataNodes: Store 128MB blocks with 3× replication.
- Designed for: Large sequential reads/writes (MapReduce).
- Weakness: Small files, random access, low latency (not designed for these).
I/O Patterns and File System Choice
| Workload | I/O Pattern | Best File System |
|---|---|---|
| HPC simulation (CFD, molecular dynamics) | Large checkpoint writes, parallel reads | Lustre, GPFS |
| ML training (ImageNet, web data) | Random small reads, sequential large reads | WekaFS, Lustre, GPFS |
| Big data analytics (Spark) | Sequential scan, shuffle | HDFS, CephFS |
| AI model checkpointing | Periodic large writes (10-100 GB) | Lustre, GPFS |
| Genomics pipeline | Many small files + large BAM files | GPFS, BeeGFS |
Performance Tuning
| Technique | What | Impact |
|---|---|---|
| Stripe count | Number of OSTs per file | More stripes → higher bandwidth |
| Stripe size | Bytes per OST before next | Match I/O request size |
| Client caching | Read-ahead and write-behind | Reduce metadata operations |
| Parallel I/O (MPI-IO) | Coordinated multi-process writes | Avoid lock contention |
Distributed file systems are the storage backbone of every HPC center and AI training cluster — without parallel file systems that can deliver hundreds of GB/s of aggregate bandwidth across thousands of concurrent readers, modern AI training runs would be bottlenecked by data loading rather than GPU computation, and HPC simulations would spend more time on I/O than on science.