Adversarial Training

Adversarial Training (AT) is the most effective defense against adversarial attacks — training the model on adversarial examples by solving a min-max optimization: the inner maximization finds the worst-case perturbation, and the outer minimization trains the model to correctly classify it.

AT Formulation

• Min-Max: $\min_\theta \mathbb{E}_{(x,y)} [\max_{|\delta| \leq \epsilon} L(f_\theta(x+\delta), y)]$.
• Inner Max: Use PGD (Projected Gradient Descent) to find the worst-case perturbation $\delta^*$.
• Outer Min: Update model parameters to minimize the loss on the perturbed input $x + \delta^*$.
• Epsilon: The perturbation budget $\epsilon$ defines the robustness guarantee.

Why It Matters

• Gold Standard: AT remains the most reliable defense against adversarial attacks after years of research.
• PGD-AT: Madry et al. (2018) showed that PGD adversarial training provides strong empirical robustness.
• Cost: AT is ~3-10× more expensive than standard training (requires PGD attack at each training step).

Adversarial Training is training on the hardest examples — building robustness by training the model to correctly classify worst-case adversarial perturbations.