Free Adversarial Training

Free Adversarial Training is a method that simultaneous updates both the model parameters and the adversarial perturbation in each gradient computation — reusing the same backward pass for both adversarial example generation and model weight update, making adversarial training essentially "free" in computational cost.

How Free AT Works

• Shared Gradient: Compute the gradient $\nabla_{x,\theta} L(f_\theta(x+\delta), y)$ — gradient w.r.t. both input AND parameters.
• Simultaneous Update: Use the gradient to update $\delta$ (for generating adversarial examples) and $\theta$ (for training) in the same step.
• Replay: Repeat $m$ times on the same minibatch, accumulating perturbation $\delta$ across replays.
• Cost: Total forward-backward passes = $m \times$ standard training (choose $m = 4-8$ for $\approx$ PGD-7 robustness).

Why It Matters

• Computational Free Lunch: Adversarial perturbation is generated "for free" using the same gradient as weight updates.
• Practical: Achieves near-PGD-AT robustness at a fraction of the compute cost.
• Memory Efficient: No need to store separate perturbation gradients — reuses the same computation.

Free AT is two-for-one gradient computation — generating adversarial examples and training the model with a single shared backward pass.