Exascale Programming Model Kokkos Raja

Exascale Programming Models are the software abstractions and runtime systems that enable scientists to express parallelism across the millions of heterogeneous processing units (CPUs + GPUs) of exascale supercomputers — addressing the fundamental challenge that no single programming model can simultaneously provide portability across diverse hardware (Intel, AMD, NVIDIA GPUs; ARM/x86/POWER CPUs), performance approaching hardware limits, and productivity for domain scientists with limited systems expertise.

The Exascale Programming Challenge

Frontier's 74,000 nodes × 4 AMD MI250X GPUs × 2 GCDs = 592,000 GPU devices + 74,000 CPU sockets. Programming this requires:
- Expressing node-level GPU parallelism (hundreds of thousands of threads).
- Expressing inter-node communication (MPI over InfiniBand/Slingshot).
- Handling heterogeneous memory (GPU HBM + CPU DRAM + NVMe burst buffer).
- Achieving portability: same code should run on Frontier (AMD), Aurora (Intel), and Summit (NVIDIA) successors.

MPI+X Hybrid Programming

The dominant production model:
- MPI between nodes (or between CPU sockets): message passing for distributed memory.
- X within a node: OpenMP (CPU threads), CUDA/HIP (GPU), OpenMP target (offload).
- MPI+CUDA: each rank owns one GPU, CUDA kernels for GPU work, MPI for inter-node. Most HPC applications today.
- MPI+OpenMP: each rank spawns OMP threads for socket-level parallelism. Used in legacy Fortran/C++ codes.
- Challenge: MPI and GPU runtime both use PCIe/NVLink — coordination needed for GPU-aware MPI (NVIDIA NVSHMEM, ROCm MPI).

Performance Portability Libraries

• Kokkos (Sandia/SNL): C++ abstraction for execution spaces (CUDA, HIP, OpenMP, SYCL) and memory spaces. View data structure (N-D array). ``parallel_for``, ``parallel_reduce``, ``parallel_scan`` policies. Used in Trilinos, LAMMPS, Albany.
• RAJA (LLNL): loop abstraction (forall, kernel), execution policies as template parameters. CHAI for memory management. Used in LLNL production codes.
• OpenMP target: standard (no library required), improving with compilers (GCC, Clang, CCE). Simpler for incremental GPU offloading.
• SYCL/DPC++: Intel's standard-based portability (compiles to CUDA, HIP, OpenCL via backends).

PGAS Languages

Partitioned Global Address Space: global memory view with local/remote distinction:
- Chapel (HPE Cray): domain parallelism (``forall``, ``coforall``), data parallelism (domains and distributions), built-in locale model for NUMA-awareness. Used in HPCC benchmark (STREAM-triad variant).
- UPC++ (C++): task-based with futures, one-sided RMA, RPCs for active messages. Used in genomics (ELBA, HipMer) and chemistry (NWChem port).
- OpenSHMEM: symmetric heap + one-sided puts/gets, POSIX-compliant, used in Cray SHMEM implementations.

Exascale Computing Project (ECP)

DOE initiative (2016-2023, $1.8B):
- 24 application projects (WarpX, ExaSMR, CANDLE, E4S).
- 6 software technology projects (Kokkos, RAJA, LLVM, OpenMPI, Trilinos, AMReX).
- E4S (Extreme-scale Scientific Software Stack): curated, tested software stack for exascale.
- Result: Frontier achieved 1.1 ExaFLOPS with production scientific codes.

Exascale Programming Models are the crucial software foundation that translates theoretical hardware capability into practical scientific computation — the abstractions, compilers, runtimes, and libraries that allow astrophysicists, climate scientists, and nuclear engineers to harness a million GPU cores without becoming GPU programming experts, making exascale supercomputing accessible to the scientific community that needs it most.