Federated Learning
Federated Learning is the distributed machine learning paradigm where models are trained across many decentralized devices (phones, hospitals, banks) without raw data ever leaving the local device — enabling collaborative model improvement while preserving data privacy, regulatory compliance (GDPR/HIPAA), and data sovereignty, with the central server only receiving model updates rather than sensitive user data.
How Federated Learning Works (FedAvg)
1. Server distributes current global model weights to selected client devices.
2. Clients train locally on their private data for E epochs (typically 1-5).
3. Clients send model updates (weight deltas or gradients) back to server.
4. Server aggregates updates: $w_{global}^{t+1} = \sum_{k=1}^{K} \frac{n_k}{n} w_k^{t+1}$.
- Weighted average by number of local samples per client.
5. Repeat for multiple communication rounds until convergence.
Key Challenges
| Challenge | Description | Mitigation |
|---|---|---|
| Non-IID data | Clients have different data distributions | FedProx, SCAFFOLD, personalization |
| Communication cost | Model updates are large, networks are slow | Gradient compression, quantization |
| Stragglers | Some devices are slower than others | Async aggregation, client sampling |
| Privacy leakage | Gradients can reveal information about data | Differential privacy, secure aggregation |
| Heterogeneous devices | Different compute/memory capabilities | Adaptive model sizes, knowledge distillation |
Non-IID Problem (The Core Challenge)
- IID (Independent and Identically Distributed): Each client has representative sample of global data.
- Non-IID (reality): User A has mostly cat photos, User B has mostly food photos.
- Non-IID causes: Client models diverge → averaging produces poor global model.
- Solutions: FedProx (proximity regularization), SCAFFOLD (variance reduction), local fine-tuning.
Privacy Enhancements
- Secure Aggregation: Cryptographic protocol ensures server sees only the aggregate update, not individual client updates.
- Differential Privacy: Add calibrated noise to client updates → formal privacy guarantee (ε-DP).
- Trade-off: More privacy (smaller ε) → more noise → lower model accuracy.
- Trusted Execution Environments: Run aggregation in secure enclaves (SGX, TrustZone).
Real-World Deployments
- Google Gboard: Next-word prediction trained on-device via federated learning.
- Apple: Siri improvement, QuickType suggestions — federated with differential privacy.
- Healthcare: Hospital networks training diagnostic models without sharing patient data.
- Financial: Banks collaboratively detecting fraud without sharing transaction records.
Federated learning is the enabling technology for privacy-preserving AI at scale — as data privacy regulations tighten globally and data remains the most sensitive asset organizations hold, federated learning provides the only viable path for collaborative model training without centralized data collection.