Checkpoint Restart Fault Tolerance

Checkpoint/Restart and Fault Tolerance enable resilience against hardware failures in long-running HPC simulations through periodic application state snapshots, essential for exascale computing where mean-time-between-failures measured in hours.

System-Level vs Application-Level Checkpointing

• System-Level (DMTCP): Dungeon Master of the Totality of Cells library. Captures entire process state (memory, open files, network sockets). Transparent to application.
• Advantages of System-Level: No code modification required. Works with legacy applications. Automatic recovery without application awareness.
• Disadvantages: Large checkpoint size (all memory including unused pages). Slower than selective application checkpoint.
• Application-Level Checkpointing: Application explicitly saves necessary state (not entire memory). Selective, optimized checkpoints. Requires code instrumentation.

SCR (Scalable Checkpoint/Restart) Library

• SCR Design: Leverages node-local storage (NVMe, RAID) for fast checkpoints. Metadata distributed across multiple nodes for reliability.
• Checkpoint Stages: Write checkpoint locally (fast, 100 GB/s NVMe). Metadata distributed synchronously (ensures recovery info available).
• Recovery: Upon failure, restart from latest checkpoint on local storage or rebuilt from distributed metadata.
• Scalability: Checkpoint time ~1-10 seconds for 100k node systems (vs minutes for single shared storage). Enables frequent checkpoint (every 100-1000 iterations).

Checkpoint Interval Optimization (Young's Formula)

• Optimal Interval: T_checkpoint = sqrt(2 × MTBF × T_checkpoint_time). Balances checkpoint overhead vs recovery loss.
• MTBF (Mean Time Between Failure): System reliability metric. Decreases with scale. 100k nodes, 24hr MTBF typical (T_checkpoint ~10 minutes).
• Young's Formula Derivation: Minimize: (work lost in failure + checkpoint time) / total work. Optimal checkpoint interval sqrt(2 × MTBF × Tchkpt).
• Practical Example: MTBF = 1 day, checkpoint time = 10 min. Optimal interval = sqrt(2 × 86400 × 600) ≈ 32,000 sec ≈ 9 hours. Trade-off: frequent checkpoint (overhead) vs longer recovery.

HDF5 and Parallel HDF5 for Checkpoint Data

• HDF5 (Hierarchical Data Format 5): Self-describing binary format. Metadata + data coexist. Ideal for large scientific datasets.
• Parallel HDF5 (pHDF5): Collective I/O by all MPI ranks to single file. MPIIO underneath enables all-ranks-to-single-file write in parallel.
• Checkpoint Structure: Groups per timestep (e.g., /timestep_1000/), datasets for arrays/fields. Metadata (simulation time, iteration count) stored as HDF5 attributes.
• I/O Performance: pHDF5 achieves 100-500 GB/s aggregate throughput on large clusters (vs 10-50 GB/s single-file write).

In-Memory Checkpointing

• Memory-to-Memory Checkpoints: Store checkpoints in node-local DRAM (redundant copy) instead of persistent storage. Faster than disk I/O.
• Redundancy Pattern: Checkpoint of rank R stored on different rank (cross-node mirroring). Single node failure → recover from mirror.
• Trade-off: Extra memory overhead (~2× for mirroring). Resilience to single node failure only; multiple simultaneous failures unrecoverable.
• Hybrid: Disk checkpoint every N iterations (slow but persistent), in-memory checkpoints between disk checkpoints (fast). Best of both.

Silent Data Corruption (SDC) Detection

• Silent Error Threat: Bit flips from cosmic rays, manufacturing defects undetected. Undetected errors cause incorrect results (scientific validity compromised).
• Detection via Redundancy: Dual computation (compute twice, compare results). Bit flips detected via mismatch. Redundancy overhead ~100% (2x compute).
• Application-Level Detection: Sanity checks on computed quantities. Example: conservation laws (energy, mass), bounds checks (physical reasonableness).
• Lightweight Checking: Checksum computation (CRC over data). Detects most errors, overhead ~10% (single checksum pass).

Exascale Fault Tolerance Challenges

• Failure Rate Scaling: Mean-time-between-failure inversely proportional to system size. 1 exaflop = 10^18 FLOP/s = 1000 PF systems (combined). MTBF ~30 minutes at exascale.
• Checkpoint Overhead: Checkpoint time scales with data size. Exascale system checkpoint ~100GB-1TB (Checkpoint time = 10-100 sec @ 10TB/s interconnect).
• Optimal Interval: sqrt(2 × 1800s × 50s) ≈ 450 seconds. Checkpoint every ~7 minutes optimal. Overhead: 50/450 ≈ 11% lost to checkpointing.
• Novel Approaches: Algorithmic redundancy (encode computation, tolerate errors), lossy compression (lower checkpoint precision for reduced size), in-situ analytics (checkpoint only critical outputs).

Recovery Mechanisms and Rollback

• Checkpoint-Restart Workflow: Upon failure detected, job stopped, latest checkpoint loaded, simulation resumed from checkpoint.
• Rollback Logic: Simulation time set to checkpoint timestamp. Iteration counters reset. Environment variables, pseudo-random state restored.
• Data Validity: Post-recovery, simulation continues as if failure never occurred. Transparent recovery (from application perspective).
• Checkpoint Frequency Trade-off: Frequent checkpoint = low rollback loss, high overhead. Sparse checkpoint = low overhead, high rollback loss. Optimized balance.