Similarity-Preserving Distillation

Similarity-Preserving Distillation is a knowledge distillation method that trains the student to produce the same pairwise similarity matrix as the teacher — ensuring that if two inputs are similar according to the teacher, they remain similar according to the student.

How Does It Work?

• Similarity Matrix: For a batch of N inputs, compute the N×N similarity matrix $S_{ij} = f_i^T f_j / ( f_i \cdot f_j )$.
• Loss: Minimize the difference between teacher's and student's similarity matrices: $ S^T - S^S _F^2$.
• Batch-Level: Operates on the full batch similarity structure, not individual samples.

Why It Matters

• Manifold Preservation: Ensures the student's feature space preserves the same neighborhood structure as the teacher.
• Architecture Agnostic: Works regardless of dimension mismatch between teacher and student (similarity is always N×N).
• Complementary: Can be combined with standard KD loss for improved performance.

Similarity-Preserving Distillation is transferring the social network of features — teaching the student which inputs should be friends (similar) and which should be strangers (dissimilar).