Manifold Mixup

Manifold Mixup is an extension of Mixup that performs interpolation in hidden layer representations rather than the input space — mixing intermediate features of the network, which creates smoother decision boundaries in the learned representation space.

How Does Manifold Mixup Work?

• Select Layer: Randomly choose a hidden layer $k$ from the network.
• Forward: Pass both input samples to layer $k$ independently.
• Mix: Interpolate the hidden representations: $\tilde{h}_k = \lambda h_k^{(i)} + (1-\lambda) h_k^{(j)}$.
• Continue: Forward the mixed representation through the remaining layers.
• Paper: Verma et al. (2019).

Why It Matters

• Better Than Input Mixup: Mixing in feature space creates more semantically meaningful combinations.
• Flatter Representations: Produces smoother, more regular hidden representations -> better generalization.
• Multi-Scale: Randomly selecting the mixing layer provides regularization at multiple abstraction levels.

Manifold Mixup is Mixup in thought-space — blending examples in the network's internal representations for deeper, more meaningful regularization.