Flops (floating Point Operations)

FLOPs (Floating Point Operations) measure computational cost for training or running neural networks. Definition: Count of floating point operations (addition, multiplication, etc.) performed. Training FLOPs: Approximately 6ND for transformer training, where N is parameters and D is tokens. Forward and backward pass. Inference FLOPs: Approximately 2N per token generated (forward pass only). PetaFLOP-days: Common unit for large training runs. GPT-3 trained with approximately 3640 petaflop-days. GPU specs: A100: 312 TFLOPS (FP16). H100: 1,979 TFLOPS (FP8). Theoretical vs achieved utilization differs. MFU (Model FLOP Utilization): Ratio of achieved to theoretical FLOPs. Good training achieves 40-60% MFU. Cost estimation: Convert FLOPs to GPU-hours, estimate costs. Helps plan training budgets. Comparison across models: Normalize by FLOPs to compare efficiency. Model A vs B at same compute. Precision matters: Lower precision (FP16, FP8) allows more FLOPs per second but may affect quality. Industry use: Standard metric for comparing computational requirements across papers and models.