Vectorization

Vectorization is a code optimization technique that processes multiple data elements simultaneously using SIMD instructions — replacing loops with vector operations for 4-16× speedups on numerical computations.

What Is Vectorization?

• Definition: Apply operations to entire arrays instead of element-by-element.
• Hardware: Uses SIMD (Single Instruction Multiple Data) units.
• Examples: SSE, AVX (x86), NEON (ARM).
• Languages: NumPy, PyTorch, TensorFlow leverage vectorization.
• Effect: Process 4-16 elements per instruction.

Why Vectorization Matters

• Speed: 4-16× faster than scalar loops.
• NumPy/PyTorch: Already vectorized — use array operations.
• Automatic: Compilers can auto-vectorize simple loops.
• ML/AI: All tensor operations are vectorized.
• Efficiency: Better CPU/GPU utilization.

Vectorized vs Loop

# Slow: Loop
for i in range(len(a)):
    c[i] = a[i] + b[i]

# Fast: Vectorized
c = a + b  # NumPy

Best Practices

• Avoid Python loops over arrays.
• Use NumPy/PyTorch operations.
• Batch operations when possible.
• Profile to verify vectorization.

Auto-Vectorization

Compilers can vectorize: simple loops, no dependencies, predictable access.
Use: -O3, -march=native for maximum vectorization.

Vectorization is fundamental for numerical performance — always prefer array operations over loops.