Heterogeneous Computing CPU GPU

Heterogeneous Computing is the system architecture and programming paradigm that combines different processor types (CPUs, GPUs, FPGAs, NPUs, DSPs) in a single system, dispatching each computation to the processor type best suited for it — exploiting the CPU's strength in serial, branch-heavy code and the GPU's strength in massively parallel, data-parallel workloads to achieve performance and energy efficiency beyond what any single processor type can deliver.

Why Heterogeneous

No single processor architecture is optimal for all workloads:
- CPU: Fast single-thread, branch prediction, cache hierarchy, low-latency memory access. Best for: serial code, control flow, OS operations, small tasks.
- GPU: Massive throughput, thousands of cores, high memory bandwidth. Best for: data-parallel computation, matrix operations, image/signal processing.
- FPGA: Reconfigurable logic, custom pipelines, deterministic latency. Best for: streaming data processing, network functions, custom protocols.
- NPU/TPU: Matrix multiply accelerator, low-precision arithmetic. Best for: ML inference at maximum efficiency.

Programming Models

• CUDA: NVIDIA GPU-specific. Highest performance on NVIDIA hardware. Largest ecosystem, best tooling. Not portable.
• OpenCL: Open standard for heterogeneous computing. Write-once, run on CPUs, GPUs (NVIDIA, AMD, Intel), FPGAs, DSPs. Verbose API, lower abstraction than CUDA.
• SYCL: Modern C++ single-source programming for heterogeneous devices. Host and device code in the same C++ source file. Intel oneAPI DPC++ is the primary SYCL implementation. Targets Intel GPUs, NVIDIA GPUs (via plugins), FPGAs.
• HIP (AMD): AMD's GPU programming model. API-compatible with CUDA — HIPIFY tool converts CUDA code to HIP with minimal changes. Runs on AMD GPUs natively, NVIDIA GPUs via HIP-CUDA translation.
• Unified Shared Memory (USM): Modern heterogeneous programming models (SYCL, CUDA Unified Memory) provide a single address space accessible by all devices. Data migration handled by runtime or hardware page faults.

Workload Partitioning Strategies

• Offload Model: CPU is the host; GPU is the accelerator. CPU launches GPU kernels for parallel sections, processes results serially. The dominant pattern (CUDA, OpenCL). Overhead: kernel launch latency, data transfer.
• Task-Based Partitioning: Each task in a DAG is assigned to the optimal device. CPU tasks and GPU tasks execute concurrently. Runtime systems (StarPU, OmpSs) schedule tasks dynamically.
• Streaming Partition: Pipeline stages assigned to different devices. Stage 1 (preprocessing) on CPU → Stage 2 (computation) on GPU → Stage 3 (postprocessing) on CPU. Stages execute concurrently on different data batches.

Performance Considerations

• Data Transfer Overhead: PCIe: 12-32 GB/s, 1-5 μs latency. CXL: 32-64 GB/s, sub-μs. NVLink CPU-GPU: 450-900 GB/s. The cost of moving data between processors can negate the computational benefit of acceleration.
• Amdahl's Law: If 90% of the workload is GPU-acceleratable, maximum speedup is 10×, regardless of GPU performance. The remaining serial fraction on CPU limits overall speedup.
• Roofline Overlap: The optimal device depends on arithmetic intensity. Memory-bound workloads may run equally fast on CPU and GPU; compute-bound workloads see dramatic GPU acceleration.

Heterogeneous Computing is the hardware-software co-design paradigm that maximizes system-level performance by matching each computation to its ideal processor — the recognition that the diversity of real-world workloads demands a diversity of processor architectures, unified by programming models that make the heterogeneity manageable.