GPU Oom

GPU Out of Memory (OOM) errors occur when model weights, activations, or intermediate computations exceed available GPU VRAM — a common issue in ML development that requires understanding memory usage patterns and applying techniques like gradient checkpointing, mixed precision, quantization, and batch size reduction to fit models within constraints.

What Is GPU OOM?

  • Error: "CUDA out of memory" or "RuntimeError: CUDA error: out of memory."
  • Cause: GPU memory (VRAM) exhausted by model/data/activations.
  • Context: Training uses more memory than inference.
  • Resolution: Reduce memory usage or increase available VRAM.

Why OOM Happens

  • Model Weights: Large models need gigabytes for parameters.
  • Activations: Saved for backpropagation during training.
  • Optimizer States: Adam stores 2x parameters in memory.
  • Gradients: Same size as parameters.
  • KV Cache: For inference, grows with sequence length.
  • Batch Size: More samples = more memory.

Diagnosis

Check Current Usage:

# Current GPU memory
nvidia-smi

# Real-time monitoring
watch -n1 nvidia-smi

# Detailed per-process
nvidia-smi pmon -s m

Python Memory Tracking:

import torch

# Check memory usage
print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
print(f"Cached: {torch.cuda.memory_reserved() / 1e9:.2f} GB")

# Get detailed snapshot
torch.cuda.memory_summary()

Memory Estimation:

GPU OOM — Debugging Out-of-Memory Errors CUDA out of memory: understand what eats VRAM, then reduce it systematically without sacrificing quality What Consumes GPU Memory (Training a 7B model on A100 80GB) Model weights: 14 GB (FP16) — 2 bytes × 7B Optimizer states: 28 GB (Adam: 4× weights) Gradients: 14 GB (FP16) Activations: variable Total: 56+ GB (exceeds A100 40GB!) + KV cache (inference) or activations (training) + CUDA context overhead (~1-2 GB) Fragmentation can waste 10-20% of free memory Training OOM Fixes (in order) 1. Reduce batch size simplest fix, use grad accumulation 2. Gradient checkpointing recompute activations (saves 60%) 3. Mixed precision (FP16/BF16) halve memory, use loss scaling 4. DeepSpeed ZeRO (1/2/3) shard optimizer/grad/params 5. LoRA / QLoRA train tiny adapter, freeze base 6. CPU offload move optimizer to RAM (slower) torch.cuda.empty_cache() rarely helps (fragmentation) Inference OOM Fixes 1. Quantize (INT4/INT8) 70B → 35GB (INT4 AWQ) 2. Reduce max_seq_len KV cache ∝ seq_len × batch 3. Reduce batch size fewer concurrent requests 4. Tensor parallel split across 2+ GPUs 5. PagedAttention (vLLM) eliminate KV fragmentation 6. Smaller model 8B instead of 70B (eval first) Quick Memory Estimates (FP16) 7B model 14 GB weights train: ~56 GB (Adam) 13B model 26 GB weights train: ~104 GB 70B model 140 GB weights infer INT4: ~35 GB 405B model 810 GB weights needs 8+ H100s (TP) Rule of thumb 2 bytes × params = FP16 × 4 for Adam training OOM is a memory budget problem: know exactly what goes where, then decide what to shrink (batch, precision, or model). GPU memory is the scarce resource of the AI era — every technique above trades compute time for memory savings.

Solutions

Reduce Batch Size (First try):

# If batch_size=32 OOMs:
batch_size = 16  # Try smaller
# Or even batch_size = 1 with gradient accumulation

Gradient Accumulation (Same effective batch):

accumulation_steps = 8
for i, batch in enumerate(dataloader):
    loss = model(batch) / accumulation_steps
    loss.backward()
    
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Gradient Checkpointing (Trade compute for memory):

# PyTorch native
model.gradient_checkpointing_enable()

# Hugging Face
model = AutoModel.from_pretrained(
    "model-name",
    gradient_checkpointing=True
)

# Savings: 2-3× memory reduction
# Cost: ~20% slower training

Mixed Precision Training:

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()

for batch in dataloader:
    with autocast():  # FP16 compute
        loss = model(batch)
    
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

# Savings: ~2× memory for activations

Quantization (For inference):

# bitsandbytes 4-bit
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    "model-name",
    quantization_config=bnb_config
)

# 7B model: 14 GB → ~4 GB

Clear Cache:

# Clear unused cached memory
torch.cuda.empty_cache()

# Delete unused variables
del large_tensor
torch.cuda.empty_cache()

# Use context manager for temporary tensors
with torch.no_grad():
    # Inference without saving gradients
    output = model(input)

Memory-Efficient Techniques Summary

Technique            | Memory Savings | Trade-off
---------------------|----------------|-------------------
Smaller batch size   | Linear         | More iterations
Gradient accumulation| None (same effect)| Code complexity
Gradient checkpointing| 2-3×          | 20% slower
Mixed precision (FP16)| 2× activations| Minor precision
Quantization (INT4)  | 4× weights     | Quality varies
Flash Attention      | ~2× attention  | None
DeepSpeed ZeRO       | Split across GPUs| Multi-GPU needed

Inference OOM

# Use vLLM for efficient inference
from vllm import LLM

llm = LLM(
    model="model-name",
    quantization="awq",  # 4-bit quantization
    gpu_memory_utilization=0.9  # Use 90% of VRAM
)

# Reduce context length if needed
llm = LLM(model="model", max_model_len=4096)

GPU OOM is the most common issue in ML development — understanding where memory goes and systematically applying reduction techniques enables running larger models on available hardware, making memory optimization skills essential for ML engineers.

Take gpu oom further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app