Online Learning

Online Learning

What is Online Learning?
Learning from streaming data one sample (or mini-batch) at a time, updating the model incrementally rather than retraining from scratch.

Online vs Batch Learning
| Aspect | Batch | Online |
|--------|-------|--------|
| Data access | Full dataset | One sample at a time |
| Training | Multiple epochs | Single pass |
| Memory | Store all data | Constant memory |
| Adaptation | Periodic retraining | Continuous updates |

Online Learning Algorithms

Stochastic Gradient Descent

def online_sgd(model, data_stream, lr=0.01):
    for sample in data_stream:
        x, y = sample
        prediction = model(x)
        loss = criterion(prediction, y)

        loss.backward()
        for param in model.parameters():
            param.data -= lr * param.grad
            param.grad.zero_()

Online Gradient Descent with Regret

# Track cumulative regret
cumulative_loss = 0
best_fixed_loss = compute_best_in_hindsight(data_stream)

for t, sample in enumerate(data_stream):
    loss = model.loss(sample)
    cumulative_loss += loss
    model.update(sample)

regret = cumulative_loss - best_fixed_loss
# Want sublinear regret: O(sqrt(T)) or O(log T)

Challenges
| Challenge | Mitigation |
|-----------|------------|
| Concept drift | Adaptive learning rates, windowing |
| Catastrophic forgetting | Experience replay |
| Noisy samples | Robust loss functions |
| Non-stationarity | Discount old data |

Concept Drift Detection

class DriftDetector:
    def __init__(self, window_size=100, threshold=0.05):
        self.window = deque(maxlen=window_size)
        self.threshold = threshold

    def update(self, error):
        self.window.append(error)

        if len(self.window) == self.window.maxlen:
            recent = list(self.window)[-50:]
            old = list(self.window)[:50]

            if mean(recent) - mean(old) > self.threshold:
                return True  # Drift detected
        return False

Use Cases
| Use Case | Examples |
|----------|----------|
| Recommendations | User preferences evolve |
| Fraud detection | Attack patterns change |
| NLP | Language trends shift |
| Finance | Market conditions change |

Frameworks
| Framework | Features |
|-----------|----------|
| River | Python online learning |
| Vowpal Wabbit | Fast online learning |
| Flink ML | Streaming ML |

Best Practices
- Use appropriate learning rate schedules
- Monitor for concept drift
- Consider data buffering for stability
- Evaluate on recent data