Multi-Task Learning

Multi-Task Learning (MTL) is a training paradigm where a single model is trained simultaneously on multiple related tasks — leveraging shared representations to improve generalization, reduce overfitting, and reduce the total number of parameters compared to separate task-specific models.

Core Principle

• Inductive transfer: Learning auxiliary tasks acts as regularization for the primary task.
• Shared features: Tasks share a common backbone; task-specific heads branch off.
• More data effective: Combining data from multiple tasks provides more training signal.

MTL Architectures

Hard Parameter Sharing:
- Shared encoder layers + separate output heads per task.
- Most common: BERT fine-tuned with [CLS] → different linear heads for classification, NER, QA.
- Risk: Task interference — conflicting gradients can hurt individual tasks.

Soft Parameter Sharing:
- Each task has its own model, but parameters are regularized to be similar.
- Cross-stitch networks: Learn linear combination of feature maps across tasks.
- Sluice networks: Generalization of cross-stitch with learnable sharing.

Task Balancing Challenges

• Dominant task problem: High-loss task dominates gradient → others undertrained.
• Solutions:
• Uncertainty weighting (Kendall et al.): Weight losses by learned task uncertainty.
• GradNorm: Normalize gradient magnitudes across tasks.
• PCGrad: Project conflicting task gradients to prevent interference.

MTL in Foundation Models

• GPT/T5: Implicitly multi-task — trained on diverse text → encodes multi-task knowledge.
• Gemini: Natively multi-modal — same model for text, image, audio.
• Whisper: Multi-task speech — transcription, translation, language ID, timestamps.

When MTL Helps

• Tasks share low-level features (edge detection → object detection, grammar → semantics).
• Limited data for primary task — auxiliary tasks provide regularization.
• Tasks have complementary data distributions.

Multi-task learning is a powerful regularization and efficiency strategy — the shared backbone learns richer representations than any single task would produce, and foundation models trained on diverse tasks generalize far better than narrow specialists on real-world distributions.