Fft Fast Fourier Transform Parallel

Parallel FFT: Cooley-Tukey Decimation and GPU/Distributed Implementation — achieving O(N log N) complexity across scales

The Fast Fourier Transform (FFT) is a cornerstone algorithm for signal processing, scientific computing, and machine learning inference. Cooley-Tukey decimation-in-time and decimation-in-frequency algorithms reduce naive O(N²) DFT computation to O(N log N) through recursive decomposition and reuse of twiddle factors via a butterfly computation graph.

Butterfly Network and GPU FFT

The FFT computation decomposes into log₂(N) stages, each applying butterfly operations (two inputs, two outputs with single twiddle multiplication). Butterflies exhibit natural parallelism: independent butterflies within each stage execute concurrently, with synchronization between stages. GPU FFT libraries like cuFFT provide optimized kernels for single-GPU transform and batch FFT processing. Multi-GPU FFT via cuFFTXt distributes 3D FFT data across GPUs, decomposing along trailing dimensions (pencil vs slab decomposition strategies) to minimize all-to-all transposes.

Distributed FFT with MPI

Distributed FFT libraries (PFFT, FFTW-MPI) decompose N-D transforms into sequences of 1D FFTs and global all-to-all transpose communication. 3D FFT decomposes into z-direction FFT → all-to-all transpose → y-direction FFT → all-to-all transpose → x-direction FFT. Pencil decomposition (1D or 2D slices) reduces communication volume but requires complex indexing. Communication overlapping with computation hides network latency.

Implementation Details

Bit-reversal permutation reorganizes input/output for in-place computation, often executed as a separate preprocessing step to maximize cache reuse during butterfly stages. Out-of-place implementations trade memory for reduced synchronization overhead. Twiddle factor caching in GPU shared memory and texture cache significantly reduces arithmetic intensity. Radix-4 and higher-radix variants reduce memory transactions at the cost of increased arithmetic and register pressure.