Epoch

Epoch, Batch, and Iteration are the fundamental time-keeping units of neural network training — defining how training data is organized, processed, and used to update model parameters. Understanding their relationship is essential for configuring training runs, interpreting loss curves, setting learning rate schedules, and comparing results across different research papers and implementations.

Core Definitions

Epoch — one complete pass through the entire training dataset.
- Every training sample has been seen exactly once
- After each epoch, the dataset is typically shuffled before the next pass
- Most vision models train for tens to hundreds of epochs; ResNet-50 on ImageNet trains for 90 epochs
- LLM pre-training often completes well under 1 epoch (the dataset is larger than the compute budget can exhaust)

Mini-batch (Batch) — a subset of training samples processed together in a single forward-backward pass.
- All samples in the batch are processed in parallel on the GPU
- The loss is averaged over all samples in the batch before backpropagation
- Typical sizes: 32, 64, 128, 256 for vision; 2M-16M tokens for LLM training
- Smaller batches: more gradient noise, potentially better generalization, less parallelism
- Larger batches: less noise, more stable training, better hardware utilization

Iteration (Step) — one weight update from one mini-batch.
- One iteration = one forward pass + one backward pass + one optimizer step
- This is the fundamental unit of training time: most training logs report metrics per step
- Learning rate schedulers count steps, not epochs

The Mathematical Relationship

Example: ImageNet (1.28M images), batch size 256, 90 epochs:
- Iterations per epoch: $1{,}280{,}000 / 256 = 5{,}000$
- Total iterations: $90 \times 5{,}000 = 450{,}000$

Training Loop Structure

for epoch in range(num_epochs):           # outer loop: dataset passes
    dataloader.shuffle()                   # randomize order each epoch
    for batch_x, batch_y in dataloader:   # inner loop: mini-batches
        optimizer.zero_grad()              # clear previous gradients
        predictions = model(batch_x)       # forward pass
        loss = criterion(predictions, batch_y)  # compute loss
        loss.backward()                    # backpropagate gradients
        optimizer.step()                   # update weights
        iteration += 1                     # count step
    validate(model)                        # evaluate after each epoch

This triple structure — dataset → epoch → batch → iteration — is the heartbeat of all neural network training.

LLM Pre-training: Token-Based Counting

Large language models redefine these concepts around tokens rather than samples:

• Token batch: Global batch size measured in tokens, not samples. LLaMA 3 used 4M tokens/batch; GPT-3 used 3.2M tokens/batch
• Training tokens: Total tokens processed = global batch size × total steps. LLaMA 3.1 was trained on 15 trillion tokens.
• Epoch: LLM training rarely completes even 1 epoch — the Chinchilla paper shows that for compute-optimal training, models should be trained on 20× more tokens than parameters, which for a 70B model means 1.4T tokens — most datasets aren't that large, so epochs are rare

Learning Rate Scheduling and Steps

Learning rate schedules operate on steps, not epochs:

Standard LLM training: 1-2% warmup steps, then cosine decay for remainder.

Shuffling and Data Order

Shuffle training data before each epoch:
- Prevents the model from learning spurious order-dependent patterns
- Ensures different batches each epoch, improving sample diversity
- For LLM training: documents are shuffled and concatenated (then split into fixed-length sequences), so epoch boundaries are approximate

Gradient Accumulation and Virtual Batch Size

When GPU memory limits batch size, gradient accumulation enables larger virtual (effective) batches:

One iteration in terms of weight updates corresponds to $N_{\text{accum}}$ forward-backward micro-steps. Training logs typically count optimizer steps (weight updates), not micro-steps.

Practical Guidance

• How many epochs for my task?
• Image classification (from scratch): 90-300 epochs
• Fine-tuning a pre-trained vision model: 10-30 epochs
• SFT fine-tuning an LLM: 1-3 epochs over instruction data
• LLM pre-training: <1 epoch (token-budget limited)

• How should I pick batch size?
• Use the largest batch that fits in memory
• Scale learning rate proportionally: $\eta \propto \sqrt{B}$ (square root rule) or $\eta \propto B$ (linear scaling for SGD)
• For LLMs: target 1M-16M tokens/batch for stable training

• Should I care about epochs or steps?
• For fixed datasets: epochs make sense (you know when data is exhausted)
• For streaming/large-scale training: steps are the natural unit (you set a compute budget)
• Learning rate schedules always use steps
• Early stopping monitors validation metrics after each epoch

Epoch, batch, and iteration are the vocabulary of training — every training script, research paper, and debugging conversation uses these terms, and their precise relationship determines how learning rate, regularization, and compute budget interact.