Federated Learning Privacy

Federated Learning is the distributed machine learning paradigm where multiple clients (devices or organizations) collaboratively train a shared model without exchanging raw data — each client trains on local data and shares only model updates (gradients or weights) with a central server, preserving data privacy while leveraging the collective knowledge of all participants.

Federated Averaging (FedAvg):
- Algorithm: server distributes global model to selected clients → each client performs E epochs of local SGD on its private data → clients send model updates to server → server averages updates weighted by local dataset size → repeat
- Communication Rounds: typical convergence requires 100-1000 communication rounds — each round involves model distribution (server→clients) and update collection (clients→server); communication of full model weights dominates system cost
- Client Selection: each round samples a fraction C of clients (typically 1-10%) — random selection provides unbiased gradient estimates; clients with more data may be preferentially selected for faster convergence
- Local Epochs: more local epochs (E>1) reduce communication rounds but increase divergence between client models — client drift accumulates when local data distributions differ significantly from the global distribution

Data Heterogeneity Challenges:
- Non-IID Data: client data distributions are typically non-identical — some clients may have only certain classes or heavily skewed distributions; non-IID data causes client model divergence and slower convergence
- Label Skew: different clients have different label distributions — solutions: sharing a small global dataset, FedProx with proximal term preventing excessive divergence from global model, SCAFFOLD using control variates for variance reduction
- Feature Skew: same labels but different feature distributions across clients — different lighting conditions, camera angles, or demographics; domain adaptation techniques help bridge feature gaps
- Quantity Skew: vastly different dataset sizes across clients — small-data clients may overfit locally; weighted averaging mitigates by giving less weight to small datasets

Privacy and Security:
- Privacy Guarantees: raw data never leaves the client — but model updates can leak information; gradient inversion attacks can reconstruct training images from shared gradients
- Differential Privacy: add calibrated noise to model updates before sharing — provides mathematical privacy guarantee (ε-differential privacy); traded against model accuracy; typical ε=1-10 for practical use
- Secure Aggregation: cryptographic protocol ensures server only sees the aggregate of all client updates, not individual contributions — protects against honest-but-curious server; adds 2-5× communication overhead
- Byzantine Resilience: robust aggregation methods (trimmed mean, Krum, median) tolerate malicious clients submitting poisoned updates — critical for open participation scenarios

Federated learning enables AI model training in privacy-sensitive domains (healthcare, finance, mobile) where data cannot be centralized — organizations like Google (Gboard), Apple (Siri), and hospitals collaborating on medical AI already deploy federated learning in production systems.