Virtual Adversarial Training

VAT (Virtual Adversarial Training) is a semi-supervised regularization technique that computes the worst-case perturbation to inputs and penalizes the model for changing its predictions — enforcing local smoothness of the output distribution around both labeled and unlabeled data.

How Does VAT Work?

• Find Adversarial Direction: $r_{adv} = argmax_{ r \leq \epsilon} \text{KL}(p(y|x) p(y|x+r))$ (direction that maximally changes predictions).
• Power Iteration: Approximate $r_{adv}$ using 1-2 steps of power iteration (efficient).
• Loss: $\mathcal{L}_{VAT} = \text{KL}(p(y|x) p(y|x+r_{adv}))$ (penalize prediction change under worst-case perturbation).
• Paper: Miyato et al. (2018).

Why It Matters

• No Labels Needed: The VAT loss is computed without labels -> can be applied to unlabeled data.
• Local Smoothness: Enforces that predictions are robust to small input perturbations.
• Universal: Works for any model differentiable with respect to its input (images, text embeddings, etc.).

VAT is adversarial robustness as regularization — finding and defending against worst-case perturbations to enforce smooth, confident predictions.