Cross-Modal Distillation

Cross-Modal Distillation is a knowledge distillation technique that transfers knowledge from one modality to another — for example, transferring visual knowledge from an image model to a depth-only model, or from a text model to a speech model, enabling inference on a single modality using knowledge from a richer one.

How Does Cross-Modal Distillation Work?

• Setup: Teacher trained on modality A (e.g., RGB images). Student trained on modality B (e.g., depth maps).
• Transfer: Student learns to mimic teacher's representations when both see the same scene from different modalities.
• Paired Data: Requires paired multi-modal data during training (e.g., RGB + depth pairs).

Why It Matters

• Sensor Reduction: Deploy with only a cheap/available sensor (depth camera) while benefiting from knowledge learned on an expensive sensor (RGB camera).
• Multimodal AI: Enables models that operate on one modality to benefit from another modality's knowledge.
• Applications: Robotics (RGB teacher -> depth student), medical imaging (MRI teacher -> ultrasound student).

Cross-Modal Distillation is knowledge translation between senses — teaching a model that can only see depth to understand the world as if it could also see color.