Spmd Programming

SPMD (Single Program Multiple Data) is the dominant parallel programming model where all processors execute the same program but operate on different portions of data, using their processor ID to determine which data to process — forming the foundation of MPI programming, GPU computing (CUDA), and virtually all large-scale parallel applications, where a single codebase scales from 1 to millions of processors by parameterizing behavior on rank or thread ID rather than writing separate programs for each processor.

SPMD Concept

Same program, different data:
 Rank 0: process(data[0:250])    ← Same code
 Rank 1: process(data[250:500])  ← Different data partition
 Rank 2: process(data[500:750])  ← Different data partition
 Rank 3: process(data[750:1000]) ← Different data partition

SPMD vs. Other Models

MPI SPMD Pattern

int main(int argc, char **argv) {
    MPI_Init(&argc, &argv);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    
    // Same code, different behavior based on rank
    int chunk = N / size;
    int start = rank * chunk;
    int end = start + chunk;
    
    // Each rank processes its portion
    double local_sum = 0;
    for (int i = start; i < end; i++)
        local_sum += compute(data[i]);
    
    // Collective: combine results
    double global_sum;
    MPI_Reduce(&local_sum, &global_sum, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
    
    MPI_Finalize();
}

CUDA as SPMD

// Every thread runs same kernel, different threadIdx
__global__ void vector_add(float *a, float *b, float *c, int n) {
    int id = blockIdx.x * blockDim.x + threadIdx.x;  // Unique ID
    if (id < n)
        c[id] = a[id] + b[id];  // Same operation, different element
}
// Launch: 10000 threads all run vector_add but on different indices

Bulk Synchronous Parallel (BSP)

Superstep 1:   [Compute]  →  [Communicate]  →  [Barrier]
 Superstep 2:   [Compute]  →  [Communicate]  →  [Barrier]
 Superstep 3:   [Compute]  →  [Communicate]  →  [Barrier]

• BSP = SPMD + explicit supersteps.
• Each superstep: Local computation → communication → global barrier.
• Predictable performance: Cost = max(compute) + max(communication) + barrier.
• Used by: Google Pregel (graph processing), Apache Giraph, BSPlib.

SPMD Advantages

SPMD + Data Parallelism in ML

• Distributed data parallel (DDP): Each GPU runs same model on different mini-batch.
• Same forward pass, same backward pass, different data → classic SPMD.
• AllReduce (gradient sync) = BSP barrier between iterations.
• FSDP: SPMD where each rank holds different model shard.

SPMD is the programming model that makes large-scale parallelism tractable — by writing a single program that adapts its behavior based on processor identity, SPMD eliminates the complexity of coordinating different programs while naturally expressing data decomposition, making it the universal foundation that underlies MPI applications on supercomputers, CUDA kernels on GPUs, and distributed training frameworks in machine learning.