Iteration

Training Terminology: Epochs, Batches, Iterations

Definitions

Batch
A subset of training examples processed together:

batch_size = 32  # Process 32 examples at once

Iteration (Step)
One forward + backward pass on a single batch:

1 iteration = process 1 batch = 1 gradient update

Epoch
One complete pass through the entire training dataset:

1 epoch = dataset_size / batch_size iterations

Example Calculation

Dataset: 10,000 examples
Batch size: 32
Iterations per epoch: 10,000 / 32 ≈ 312

Training for 3 epochs = 3 × 312 = 936 total iterations

Effective Batch Size

Gradient Accumulation
Process more examples before updating weights:

accumulation_steps = 4
effective_batch_size = batch_size × accumulation_steps
# 32 × 4 = 128 effective batch size

Why use it:
- Fit larger effective batches on limited GPU memory
- More stable gradients

Distributed Training
With multiple GPUs:

global_batch_size = batch_size × num_gpus × accumulation_steps

LLM Training Scale

Pretraining
| Model | Tokens | Epochs | Notes |
|-------|--------|--------|-------|
| GPT-3 | 300B | <1 | Never repeats data |
| Llama 2 | 2T | ~1 | Some repetition |
| Llama 3 | 15T | ~4 on some data | Selective repetition |

Fine-Tuning
| Method | Typical Epochs |
|--------|----------------|
| SFT | 1-3 |
| LoRA | 1-5 |
| Full fine-tuning | 1-3 |

More epochs risk overfitting on small datasets.

Training Code Example

num_epochs = 3
batch_size = 32
accumulation_steps = 4

for epoch in range(num_epochs):
    for i, batch in enumerate(dataloader):
        # Forward pass
        loss = model(batch)
        loss = loss / accumulation_steps
        loss.backward()

        # Update only every N steps
        if (i + 1) % accumulation_steps == 0:
            optimizer.step()
            optimizer.zero_grad()

    print(f"Completed epoch {epoch + 1}")

Monitoring Progress

Step 1000: loss=2.34, lr=0.0001
Step 2000: loss=1.87, lr=0.0001
Epoch 1/3 complete
...