Mini-Batch Online Learning
Mini-batch online learning is a hybrid approach that combines aspects of batch and online learning by updating the model with small batches of streaming data rather than one example at a time or waiting for the complete dataset. It provides a practical middle ground for real-world systems.
How It Works
- Accumulate: Collect a small batch of new examples (e.g., 32–256 examples).
- Compute Gradients: Calculate the gradient of the loss across the mini-batch.
- Update Model: Apply the gradient update to model parameters.
- Continue: Move to the next mini-batch as data arrives.
Why Mini-Batches Instead of Single Examples?
- Gradient Stability: Single-example gradients are very noisy — they point in unpredictable directions. Mini-batch gradients average over multiple examples, providing a much more reliable update direction.
- Hardware Efficiency: GPUs are designed for parallel computation. Processing one example at a time wastes GPU capacity. Mini-batches fill the GPU's parallel compute units.
- Learning Rate Sensitivity: Single-example updates require very small learning rates to avoid instability. Mini-batches allow larger, more effective learning rates.
Mini-Batch vs. Other Approaches
| Approach | Batch Size | Update Frequency | Gradient Quality |
|---|---|---|---|
| Full Batch | Entire dataset | Once per epoch | Best (exact gradient) |
| Mini-Batch | 32–256 | After each batch | Good (approximate gradient) |
| Online (SGD) | 1 | After each example | Noisy (stochastic) |
| Mini-Batch Online | 32–256 (streaming) | As data arrives | Good + adaptive |
Applications
- Real-Time Model Adaptation: Update recommendation models as new user interactions arrive in small batches.
- Streaming Analytics: Process log streams or sensor data in micro-batches.
- Continual Fine-Tuning: Periodically micro-fine-tune LLMs on recent data batches.
- Federated Learning: Clients compute updates on local mini-batches and share aggregated gradients.
Practical Considerations
- Batch Size Selection: Larger batches are more stable but introduce more latency before each update. Typical range: 32–256.
- Learning Rate Scheduling: Online mini-batch updates often benefit from warm-up and decay schedules.
- Validation: Periodically evaluate on a held-out set to detect degradation.
Mini-batch online learning is how most production ML systems actually operate — it balances the theoretical purity of online learning with the practical stability of batch training.